Retrieval-Augmented Generation (RAG) demos are easy to build and easy to fool yourself with. A client came to us after their internal knowledge-base chatbot, built in an afternoon with a popular no-code tool, started confidently citing policies that didn't exist. Here's the production architecture we built to replace it — one that actually admits when it doesn't know something.
Why the Original Demo Failed in Production
The demo used naive fixed-size chunking (splitting documents every 500 characters regardless of content boundaries) and a single similarity search with no reranking or confidence threshold. This produced two failure modes: retrieving irrelevant chunks that happened to share vocabulary with the query, and the LLM confidently synthesizing an answer even when nothing relevant was actually retrieved.
The Architecture We Built
1. Semantic Chunking, Not Fixed-Size
Instead of splitting text at arbitrary character counts, we chunk along semantic boundaries — section headers, paragraph breaks — with a target size window and overlap to preserve context across chunk boundaries.
2. Hybrid Retrieval
Pure vector similarity search misses exact-match queries (a policy number, a specific term). We run hybrid retrieval combining dense vector search (via Amazon OpenSearch Service with k-NN) and sparse keyword search (BM25), merging results before reranking.
3. Reranking Layer
The top-N candidates from hybrid retrieval get reranked by a cross-encoder model that scores query-document relevance more precisely than the initial retrieval step — this is the single highest-leverage addition for retrieval quality in our experience.
def retrieve_and_rerank(query: str, top_k: int = 5) -> list[dict]:
vector_results = opensearch_client.knn_search(query_embedding, k=20)
keyword_results = opensearch_client.bm25_search(query, k=20)
merged = merge_and_dedupe(vector_results, keyword_results)
reranked = cross_encoder.rerank(query, merged)
return [doc for doc in reranked if doc.score > CONFIDENCE_THRESHOLD][:top_k]
4. Confidence Threshold and Explicit "I Don't Know"
If no retrieved chunk clears a minimum relevance score, the system returns "I don't have information on this in the current knowledge base" instead of passing weak context to the LLM and letting it fill gaps with plausible-sounding fabrication.
5. Source Citation Enforcement
Every generated answer is required to reference the specific source chunk IDs used, which are then mapped back to the original document and section for the user to verify — turning the chatbot into a research assistant that shows its work, not an oracle.
Handling Document Updates
A common gap in RAG demos: what happens when the underlying documents change? We built an ingestion pipeline that watches the source document repository for changes, re-chunks and re-embeds only the modified documents, and updates the vector index incrementally rather than requiring a full reindex.
Evaluation Before Launch
Before rolling this out, we built a test set of representative questions with known-correct answers and measured retrieval precision and answer accuracy against it — a step almost universally skipped by teams that ship a RAG chatbot straight from a demo. This evaluation harness also became the regression test suite for future changes to the pipeline.
Results
The rebuilt pipeline reduced fabricated answers to near zero on the evaluation set (versus a meaningful fabrication rate in the original demo) and gave the client's support team a tool their staff actually trusted enough to use in front of customers, because every answer is traceable to a specific source document.
Building a knowledge base tool that needs to actually be trustworthy? Discuss your automation project — 907-841-8407 or contact@rutagon.com.
Frequently Asked Questions
What's the difference between a RAG demo and a production RAG system?
A demo typically uses naive fixed-size chunking, single-method retrieval, and no confidence threshold — it always generates an answer even when retrieval quality is poor. A production system adds hybrid retrieval, reranking, confidence thresholds, and citation enforcement to catch and prevent fabrication.
How do you prevent an LLM from hallucinating answers in a RAG pipeline?
The most effective technique is enforcing a minimum relevance confidence threshold on retrieved chunks — if nothing scores high enough, the system explicitly says it doesn't have the information rather than letting the LLM generate a plausible-sounding but unsupported answer.
What is reranking and why does it matter for RAG?
Reranking uses a more precise (but computationally heavier) model to re-score the top candidates from an initial retrieval pass. Since it only runs on a small candidate set rather than the entire corpus, it improves relevance significantly without the cost of running the precise model against every document.
Does RAG work with our existing document repository (SharePoint, Confluence, Google Drive)?
Yes. The ingestion layer can connect to most common document sources via their APIs, watching for changes and feeding new or updated content into the chunking and embedding pipeline automatically.
How do you measure whether a RAG system is actually accurate?
We build an evaluation set of representative questions with known-correct answers, then measure retrieval precision (did it find the right source) and answer accuracy (did the generated response correctly reflect that source) before and after any pipeline changes.