A retrieval prototype is easy to build and easy to demo. You load a folder of documents, embed them, drop a similarity search in front of a language model, and it answers questions well enough that everyone in the room nods. Then it meets real documents and real users, and the failure modes arrive all at once.
What follows is what we changed when we took a multi-tenant retrieval platform from that demo state into production. None of it is exotic. Most of it is the unglamorous work that decides whether the thing is usable.
Build the evaluation set before you build anything else
This is the one that matters, and it is almost always skipped.
Until you have a set of real questions with known-good answers, you cannot tell whether a change helped. You will swap the embedding model, feel that answers got better, and have no way to know. Every subsequent decision on this list (chunk size, retrieval strategy, reranker, prompt) is a trade-off you can only evaluate against a fixed set.
Pull the questions from somewhere real: support tickets, the search log, the internal wiki’s most-visited pages, or an hour with the people who currently answer these questions by hand. A hundred questions is enough to start. It does not need to be clean, and it does not need to be big. It needs to exist before you start tuning.
Chunking is a retrieval decision, not a preprocessing step
Splitting documents on a fixed token count is the default, and it is the default because it is easy, not because it works. It cuts tables in half, separates a heading from the paragraph it introduces, and strips the context that made a sentence meaningful.
Two things helped more than tuning the chunk size:
- Split on document structure where the format gives you one. Headings, sections and list boundaries carry meaning that a character count does not.
- Keep the parent context with the chunk. A chunk that knows which document and which section it came from can be reranked and cited properly. A bare paragraph cannot.
The chunk you retrieve and the chunk you show the model do not have to be the same thing. Retrieving precisely and then expanding to the surrounding context before generation is usually better than trying to find one chunk size that serves both.
One retrieval strategy is not enough
Pure vector search is very good at “find me something about this idea” and unreliable at “find me the document that contains this exact term”. Product codes, error strings, policy numbers and proper nouns are exactly the queries where users expect an exact match, and exactly where embeddings are weakest.
Keyword search, plain BM25, handles those cases well and has been doing so for decades. In the platform we built, retrieval could run BM25-only, semantic-only, or hybrid, and the hybrid path was the one that survived contact with real questions. It is worth building all three even if you expect to use one, because the comparison tells you what your corpus actually needs.
Reranking is the cheapest accuracy you will buy
Retrieval gets you a candidate set. Reranking decides what the model actually sees, and it is consistently the highest-yield change available.
Two approaches, and they solve different problems:
- Reciprocal rank fusion merges results from several retrieval strategies without needing scores that are comparable across them. Cheap, fast, no model to host.
- Cross-encoder rerankers score each candidate against the question directly rather than comparing pre-computed vectors. Slower and heavier, and noticeably more accurate.
We made the reranker a configuration choice rather than an architectural commitment, and supported four of them. Corpora differ enough that the right answer is not predictable in advance, which is another reason the evaluation set has to come first.
The question the user typed is rarely the query you should run
People ask short, underspecified, context-dependent questions. “What about the second one?” is meaningless without the previous turn. “What is the retention policy?” is ambiguous when four departments have one.
Planning the query before retrieving it made a larger difference than any single model change:
- Resolve the question against the conversation history so follow-ups work.
- Expand it into several keyword and semantic variants and retrieve for all of them. Different phrasings surface different documents.
- Apply a glossary. Every organisation has internal terms, acronyms and product names that appear nowhere in the model’s training data.
Exact-match caching does almost nothing; semantic caching does a lot
Caching on a hash of the question has a hit rate near zero, because nobody types the question the same way twice. Caching on the meaning of the question, by embedding it and matching against previous questions above a similarity threshold, removes a meaningful share of redundant model calls.
Set the threshold carefully and get it wrong in the safe direction. A cache that returns a subtly wrong answer to a similar-but-different question is worse than no cache, because the failure is invisible.
Let the system criticise its own answer
A single pass through retrieval and generation produces confident answers regardless of whether the retrieved context supported them. Adding a critique step that scores the draft answer against the retrieved context, then refines or retries when it falls short, catches a category of failure that no amount of retrieval tuning will.
Make the number of iterations configurable and low. This trades latency and cost for accuracy, and the right balance is different for an internal tool than for a customer-facing one.
Multi-tenancy is an isolation problem, not a filtering problem
If several customers’ documents live in one index and separation depends on a filter being applied correctly on every query path, then one missing filter is a data breach. Per-tenant indices cost more to operate and remove that entire class of bug. For anything touching customer data, we would take that trade every time.
Streaming changes how fast the system feels
A good RAG answer takes several seconds: planning, retrieval, reranking, generation, sometimes a critique pass. That is a long time to watch a spinner.
Streaming the intermediate steps over Server-Sent Events, what it is searching for, what it found, then the answer token by token, makes the same latency feel responsive rather than broken. It also makes the system debuggable in production, because you can see which stage was slow instead of guessing.
If you only do three things
- Write the evaluation set first. Everything else is guesswork without it.
- Add hybrid retrieval and a reranker. Together these are the largest accuracy gain per hour of work available.
- Plan the query before you run it. Resolve follow-ups against history and expand into variants.
None of this makes retrieval a solved problem. It does move a system from one that demos well to one you can put real users on, which is the only distinction that matters once it ships.
