Five problems to solve before a RAG system reaches production

Standing up a RAG (retrieval-augmented generation) prototype in a week is not hard any more. What is hard is that prototype still answering correctly six months later.

Across the RAG systems we have taken to production over the past two years, the same five problems keep reappearing. None of them show up in a demo; all of them get expensive in production.

1. Starting without an evaluation set

This is the most common mistake. The team asks a handful of questions, the answers look reasonable, and the system is declared to work. Then a model version changes, a document is updated, a chunk size is tweaked — and nobody can tell what broke.

A RAG system is an unmeasurable system until someone writes an evaluation set.

Our minimum bar is at least 150 question–answer pairs, verified by a domain expert, distributed like this:

• Questions answerable from a single document
• Questions requiring several documents to be combined
• Questions the corpus genuinely cannot answer
• Questions likely to confuse two similar-but-different documents

The last two categories are the most valuable. A system's ability to say "I don't know" matters as much as its ability to answer.

2. Chunk boundaries cutting through meaning

Fixed-length splitting (say, every 500 tokens) is easy to implement and usually wrong. If a table's header lands in one chunk and its rows in another, the answer will be incomplete no matter which chunk retrieval finds.

The approach we use:

• Split the document by structure first (heading, section, table, list)
• If a section is too long, split further at sentence boundaries
• Prepend the heading chain to every chunk as text
• Never split a table; break large tables into row groups and repeat the header row in each

This single change moved one client's answer accuracy from 71% to 88%.

3. Applying permissions after retrieval

A frequent design error: retrieve the top 20 most relevant chunks, then filter out what the user cannot see.

This has two problems. First, filtering may leave you with two chunks and a weak answer. Second and more serious, information can leak through ranking signals: a user can infer that a document they cannot see exists.

The permission filter belongs inside the search query, not after it. If your vector database supports metadata filtering, this is a single query.

results = client.search(
    collection_name="docs",
    query_vector=embedding,
    query_filter=Filter(
        must=[FieldCondition(key="acl", match=MatchAny(any=user.group_ids))]
    ),
    limit=8,
)

4. Letting the index fall behind the documents

Corporate documents change. A procedure is updated, a price list is refreshed, a policy is withdrawn. If the index does not track those changes, your system starts giving confidently wrong answers — which is worse than giving none.

The minimum arrangement we build:

| Component | Responsibility | | --- | --- | | Change listener | Catches updates in the source system | | Queue | Accumulates and rate-limits re-embedding work | | Version tag | Records which document version a chunk came from | | Reaper | Removes chunks of deleted documents from the index |

That last row is the most commonly skipped step. If chunks of a deleted document survive in the index, the system keeps citing a policy that no longer exists.

5. Thinking about cost afterwards

Token cost is invisible in a prototype because there is no traffic. In production, the bill is usually driven not by model choice but by context length.

Three measures we apply:

1. Narrow retrieval. Six good chunks beat twenty mediocre ones — more accurate and cheaper in most cases.
2. Cache answers. In an enterprise setting a meaningful share of questions repeat. A semantic cache keyed on the normalised question served 31% of calls for one client.
3. Match the model to the task. A small model for classification and reranking, a large one for the final answer.

Together these cut monthly token cost by 43% for that same client, with no measurable loss in accuracy.

Summary

What makes RAG hard is not the model but the engineering around it. Before taking a system to production you should have written answers to four questions:

• How do we measure accuracy?
• When does the index update after a document changes?
• At which layer are user permissions enforced?
• What happens to the bill if traffic grows tenfold?

If you cannot answer those, what you have is a demo, not a product.