You can build a retrieval-augmented chatbot in an afternoon. Load documents, chunk them, embed them, retrieve the top five, put them in a prompt. It will demo beautifully.
Then you put it in front of a support team and discover it confidently cites a policy that was superseded in 2023, and answers questions about pricing it has no business answering. Everything difficult about RAG lives between those two moments.
Chunking is a retrieval decision, not a preprocessing chore
Fixed 512-token chunks are the default in every tutorial and they are wrong for most enterprise corpora. They split tables across boundaries, orphan a heading from its content, and strip the context that made a paragraph meaningful.
What works better in practice:
- Structure-aware splitting. Chunk on document structure — headings, sections, table boundaries — not character counts.
- Contextual headers. Prefix each chunk with its document title and heading path. "Leave Policy › Maternity Leave › Eligibility" changes retrieval quality dramatically.
- Keep tables whole. A split table is worse than no table; it produces confidently wrong numbers.
- Overlap deliberately. 10–15% overlap catches answers that straddle a boundary.
Pure vector search is not enough
Embeddings are excellent at semantic similarity and poor at exact identifiers. A user asking about "policy HR-114" gets chunks about policies in general and not the one they named.
Hybrid retrieval — BM25 keyword search combined with vector similarity, fused with reciprocal rank fusion, then re-ranked by a cross-encoder — consistently outperformed either alone in our evaluations, typically by 15 to 25 percentage points of answer accuracy.
Metadata filtering is where most accuracy actually comes from
The superseded-policy failure has nothing to do with the model. It is a retrieval problem: the old document was in the index and nothing marked it stale.
Every chunk in our production systems carries effective date, expiry date, department, document status and access level. Retrieval filters on those before similarity is even considered. An expired document cannot be cited because it is never a candidate.
You need an evaluation set before you need a better prompt
This is the step teams skip, and it is the one that separates a demo from a system.
Build 100–200 question-and-answer pairs from real user questions, reviewed by someone who knows the domain. Include questions the system should refuse. Then measure on every change:
- Retrieval recall — was the correct chunk in the retrieved set at all?
- Answer faithfulness — is every claim supported by retrieved context?
- Answer relevance — does it address what was asked?
- Refusal accuracy — does it decline when it genuinely should?
Without this, prompt tuning is superstition. With it, you can tell in ten minutes whether a change helped, and you have regression protection when someone edits a prompt six months later.
Refusal is a feature, and it must be designed
An assistant that answers everything is worse than one that answers 70% of questions and clearly says "I don't have information on that — here is who to ask." The second builds trust; the first destroys it the moment someone acts on a wrong answer.
We instruct explicitly for grounding, require citations, and set a retrieval-score threshold below which the system declines rather than reaching. Users adapt to a system with visible limits far better than to one with invisible ones.
Citations are not decoration
Every answer links to the source chunk and its document. This does three things: it lets the user verify, it makes hallucination visible instead of silent, and it gives you a debugging trail when something goes wrong. Users click these links more than we expected — roughly a third of answers in one deployment.
What a realistic timeline looks like
| Phase | Duration | Output |
|---|---|---|
| Corpus audit & access model | 1–2 weeks | What exists, what is current, who may see it |
| Ingestion & chunking | 2 weeks | Indexed corpus with metadata |
| Evaluation set | 1 week, parallel | 150+ reviewed Q&A pairs |
| Retrieval tuning | 2–3 weeks | Hybrid search meeting accuracy threshold |
| Guardrails & integration | 2 weeks | Refusal policy, PII handling, deployment |
| Pilot & iteration | 4 weeks | Real usage, feedback loop, tuning |
Roughly a quarter, not an afternoon. The afternoon version is a prototype, and it is genuinely useful for deciding whether the quarter is worth spending.
The hardest part of enterprise AI is almost never the model. It is knowing which of your documents is still true.