RAG Architecture in Production: How I Built a 12-Component System

What Is RAG Architecture and Why Does It Matter?

Retrieval Augmented Generation (RAG) is a system design pattern that grounds large-language-model responses in external, up-to-date knowledge instead of relying solely on the model's training data. Rather than fine-tuning a model every time your data changes, RAG retrieves the most relevant documents at query time, injects them into the prompt context, and lets the LLM synthesise a grounded answer.

Why does this matter? Because hallucinations are the number-one deployment blocker in enterprise AI. When I started building production RAG systems for clients, accuracy on domain-specific questions hovered around 60-65%. That is not a rounding error - it means roughly one in three answers is unreliable. No engineering leader will sign off on that in a customer-facing product.

The promise of RAG architecture is simple: give the model exactly the context it needs, nothing more, nothing less. The reality of RAG architecture is anything but simple. Getting from a proof-of-concept to a production system that serves real users at enterprise scale requires deliberate decisions at every layer - from chunking and embedding to retrieval, re-ranking, and response synthesis. Over the past year I have designed, built, and shipped a 12-component RAG system that now runs in production. On its most recent public benchmark run it put the right source document in front of the model 94% of the time (239 of 253 cases), and the full results - weak suites included - are published. This article walks through every meaningful decision I made and why.

The Anatomy of a Production RAG System

Most tutorials show RAG as three boxes: embed → retrieve → generate. In production, those three boxes expand into at least a dozen interacting components. Here is the architecture I shipped, broken into its 12 core modules:

1. Document Ingestion Pipeline - normalises PDFs, HTML, Markdown, and structured data into a common intermediate representation.
2. Chunking Engine - applies context-aware, hierarchical chunking with configurable overlap and boundary detection.
3. Embedding Service - generates dense vector representations via a dedicated microservice wrapping an embedding model (currently text-embedding-3-large at 3 072 dimensions, quantised to binary for storage efficiency).
4. Vector Store (pgvector) - stores embeddings in PostgreSQL with the pgvector extension, using HNSW indexing for approximate nearest-neighbour search.
5. Keyword Index - a parallel BM25 index built on the same chunks, enabling exact-match and phrase retrieval.
6. Hybrid Retriever - merges results from the vector store and the keyword index using Reciprocal Rank Fusion (RRF).
7. Re-Ranker - a cross-encoder model that re-scores the top-k candidates on semantic relevance to the query.
8. Temporal Filter - applies time-aware weighting so recent documents outrank stale ones when temporal relevance is detected in the query.
9. Context Assembler - packs selected chunks into the prompt window, respecting token budgets and inserting source citations.
10. Response Synthesiser - the LLM call itself, with system instructions that enforce citation behaviour and confidence calibration.
11. Citation Validator - post-processes the response to verify that every claim traces back to a retrieved chunk.
12. Feedback Loop & Eval Harness - collects user feedback, runs nightly evaluation sweeps against labelled datasets, and flags accuracy regressions.

Each component has its own configuration surface, failure modes, and performance characteristics. The rest of this article dives into the ones that had the largest impact on accuracy: hybrid search, chunking strategies, temporal intelligence, and the evaluation harness that turned guesswork into measurement.

Hybrid Search: Combining Vector and Keyword Search

What is hybrid search in RAG? Hybrid search is the strategy of running both a semantic vector search and a lexical keyword search against the same corpus, then fusing the results into a single ranked list. This matters because neither approach alone is sufficient for production workloads.

Vector search (dense retrieval) excels at capturing meaning. If a user asks "How do I handle authentication failures?" and the relevant chunk uses the phrase "login error recovery flow," a good embedding model will match them. But vector search struggles with exact identifiers - model numbers, error codes, API route names. A query for ERR_CONN_REFUSED may return chunks about network connectivity in general rather than the specific error.

Keyword search (BM25) is the opposite. It is ruthlessly precise on exact terms but blind to synonyms and paraphrases. By combining both, you cover the full spectrum.

In my system, the Hybrid Retriever operates as follows:

1. The user query is simultaneously dispatched to the pgvector HNSW index and the BM25 keyword index.
2. Each index returns its top-30 results with scores.
3. Results are merged using Reciprocal Rank Fusion (RRF) with a constant k = 60, which dampens the impact of outlier scores from either source.
4. The merged top-20 are passed to the cross-encoder re-ranker for final scoring.

The weighting between vector and keyword scores is not static. I implemented a query classifier - a lightweight model that categorises incoming queries as conceptual, factual-exact, ormixed. Conceptual queries shift the RRF weight toward vector results; factual-exact queries shift toward BM25. A single static weight has to compromise between those two kinds of question; routing them separately means neither has to.

A practical implementation detail: pgvector's HNSW index in PostgreSQL 16 supports both vector_cosine_opsand vector_ip_ops. I use cosine distance for normalised embeddings, with an ef_constructionof 256 and m of 32. For BM25 I use a custom GIN index on a tsvector column, which keeps the entire retrieval stack inside Postgres - no Elasticsearch required. This dramatically simplifies deployment and operational overhead.

Chunking Strategies That Actually Work

How should you chunk documents for RAG? The answer is: it depends on your data, and you should measure rigorously. That said, here are the strategies I tested and what actually moved the needle in production.

Fixed-Size Chunking (Baseline)

The simplest approach: split text every N tokens with M tokens of overlap. I started here with 512-token chunks and 64-token overlap. It works for homogeneous prose but falls apart on structured documents like API references, legal contracts, and technical manuals where section boundaries carry semantic weight.

Recursive / Hierarchical Chunking

This is the strategy that stuck. The chunking engine first splits on document structure - headings, section breaks, code fences - then recursively splits oversized sections by paragraph and sentence boundaries. Each chunk retains a parent reference so the context assembler can pull in the surrounding section header when needed. This parent-child relationship was critical: it lets the LLM understand where a chunk lives in the document hierarchy, not just what it says.

Semantic Chunking

I experimented with embedding-based boundary detection, where you slide a window through the text and split wherever the cosine similarity between adjacent windows drops below a threshold. In theory, this produces chunks that are each internally coherent. In practice, it is slow (you must embed every window), fragile (the threshold is hard to tune globally), and in my testing it was not better enough than hierarchical chunking to justify either cost. I dropped it for production.

The Settings That Mattered

The settings the system runs with, and the reasoning behind each:

• Target chunk size: 384 tokens (not 512). Smaller chunks are more precise, and the re-ranker offsets the recall risk by scoring more candidates.
• Overlap: 48 tokens - enough to preserve sentence continuity at boundaries.
• Metadata enrichment: Every chunk carries its document title, section heading path, page number (for PDFs), and an auto-generated one-line summary used as a secondary embedding target.
• Deduplication: Near-duplicate chunks (cosine similarity > 0.97) are collapsed at ingestion time. This is surprisingly important when ingesting versioned documentation where 80% of paragraphs are unchanged between releases.

How I Measure RAG Accuracy - and the Published Results

How do you measure RAG accuracy? Separately for each stage, against questions whose right answer and right source document are known in advance. A single "accuracy" number hides which stage failed. The harness behind this system scores every case on:

• Retrieval hit - is the document that holds the answer among what the model is given?
• Pass - is the final answer graded correct, including declining when the answer is not in the corpus?
• Hallucinated citations - did the answer cite a document that was never retrieved?
• Calibration - does the system's confidence track how often it is actually right?

The most recent published run (27 April 2026, Cohere rerank-3.5) covers six suites. Four are built on public corpora I did not write; two are internal and named as such:

Two results matter more than the headline. Legal is deliberately the hardest suite: CUAD is adversarial by design, and many of its cases test whether the system correctly declines rather than answers, so its pass rate is low on purpose. And on FinanceBench the gap between 96% retrieval and 63% pass is mostly over-abstention - the right filing was retrieved, and the system declined to answer rather than risk a wrong figure. That is the right direction to fail on financial questions, and the next thing to tune.

The per-suite scorecard, methodology and per-case results are published on GitHub. The runner itself is private, so the results can be audited but not re-executed.

Zero hallucinated citations is where the design choices show. The citation validator rewrites or withholds any answer that cites a document outside the retrieved set, and the model is instructed to say "I don't have enough information" rather than guess.

Temporal Intelligence: When Your Data Has a Time Dimension

How do you handle time-sensitive data in RAG? Most RAG tutorials ignore temporality entirely. But in production, your knowledge base is not static. Policies change. Product specs get updated. Pricing shifts quarterly. If a user asks "What is the current refund policy?" and your system retrieves a chunk from 2023 alongside one from 2025, you have a problem.

The Temporal Filter component in my system works in three stages:

1. Query temporal intent detection: A lightweight classifier determines whether the query has temporal intent. Queries like "current pricing", "latest release notes", or "what changed in Q4" trigger temporal mode. Queries like"explain how HNSW indexing works" do not.
2. Timestamp-aware scoring: When temporal mode is active, each retrieved chunk receives a time-decay multiplier. I use an exponential decay function with a configurable half-life (currently 90 days). A chunk from 30 days ago gets a multiplier of ~0.79; a chunk from a year ago gets ~0.06. This multiplier is applied after the RRF fusion but before the re-ranker, so the re-ranker sees a time-adjusted candidate set.
3. Version-chain resolution: Documents that are explicit revisions of earlier documents (detected via metadata or filename patterns like policy-v3.md) trigger a version-chain lookup. Only the latest version in the chain is eligible for retrieval, and superseded versions are automatically deprioritised.

This temporal intelligence layer is what allows the system to handle evolving knowledge bases without requiring manual curation every time a document is updated. On our temporal evaluation subset (87 questions with time-sensitive answers), enabling this layer improved answer correctness from 78.2% to 94.6%.

Scaling RAG for Enterprise: Lessons from Production

What does it take to run RAG at enterprise scale? The jump from a demo to a production system serving hundreds of concurrent users surfaces problems that no tutorial warns you about. Here are the lessons that cost me the most time.

Latency Budget Management

Users expect sub-2-second time-to-first-token. My retrieval pipeline (embed query → vector search → BM25 search → RRF fusion → re-rank) takes roughly 280-400ms end-to-end. The embedding call alone is 50-80ms. To stay within budget I parallelise the vector and keyword searches, cache frequent query embeddings with a 15-minute TTL, and use a distilled cross-encoder (60M parameters instead of 400M) that re-ranks 20 candidates in under 40ms on a single GPU.

Index Maintenance

HNSW indexes in pgvector are not free. With 2.4 million chunks, the index consumes roughly 11 GB of memory. I run a nightly vacuum and reindex job during the maintenance window. For hot-path ingestion (documents that must be searchable within minutes), I use a two-tier architecture: new chunks are written to a small, flat IVFFlat index that is cheap to update, and promoted to the main HNSW index during the nightly job. Queries search both tiers and merge results.

Multi-Tenancy and Data Isolation

Enterprise clients require strict data isolation. Each tenant's chunks live in a partitioned table with row-level security enforced at the Postgres level. The pgvector HNSW index is built per partition, which means each tenant gets its own index sized to their corpus. This trades some operational complexity for hard data boundaries - a requirement for SOC 2 compliance.

Observability

Every retrieval request logs: the raw query, the classified query type, the top-10 chunk IDs with scores, the re-ranked order, the number of tokens assembled, and the latency breakdown per component. This telemetry feeds a Grafana dashboard that lets me spot accuracy regressions within hours, not weeks. When average re-ranker scores drop below a threshold, an alert fires - it usually means a recent ingestion introduced corrupted or duplicate chunks.

Common RAG Pitfalls and How to Avoid Them

What are the most common mistakes in RAG implementations? After building, reviewing, and debugging RAG systems for multiple organisations, here are the patterns I see fail most often:

1. Treating Chunking as an Afterthought

Teams spend weeks selecting embedding models and days on chunking. It should be the other way around. Chunking determines the atomic unit of retrieval - get it wrong and no amount of re-ranking will save you. A chunk that spans two unrelated topics poisons the context window. A chunk that is too small loses critical surrounding context. Invest in hierarchical, structure-aware chunking with parent references.

2. No Evaluation Harness

The second most common mistake is having no systematic way to measure accuracy. If you cannot answer "did last week's change make the system better or worse?" you are flying blind. Build a labelled eval set early - even 50 pairs is better than nothing - and run it on every configuration change.

3. Ignoring the Retrieval Step

Many teams blame the LLM when answers are wrong, but the root cause is almost always bad retrieval. If the correct chunk is not in the context window, the best model in the world cannot produce the right answer. Debug retrieval first. Log the retrieved chunks. Read them. Ask yourself: "Given only these chunks, could a human answer the question?" If not, the problem is retrieval, not generation.

4. Over-Stuffing the Context Window

More context is not always better. Including 20 marginally relevant chunks dilutes the signal and increases cost and latency. I found the sweet spot at 5-7 chunks for most queries, with the re-ranker ensuring those are the highest-quality candidates. The context assembler enforces a hard token budget and prioritises chunks with higher re-ranker scores.

5. No Feedback Loop

Production systems drift. User queries evolve. New documents introduce terminology the embedding model handles poorly. Without a feedback loop - thumbs up/down, explicit corrections, or automated eval sweeps - accuracy silently degrades. My system runs a nightly eval against the latest corpus snapshot and flags any metric that drops more than two percentage points.

RAG vs Fine-Tuning: When to Use Which

Should you use RAG or fine-tuning? This is the question I get asked most often. The answer is not either/or - they solve different problems, and in many production systems they complement each other.

Use RAG When:

• Your knowledge base changes frequently (weekly or more).
• You need source attribution and citations for every answer.
• Data access is governed by permissions - different users see different documents.
• You need to add new knowledge without retraining or waiting for a fine-tuning job.
• Accuracy on long-tail, domain-specific facts is critical (RAG retrieves the exact passage).

Use Fine-Tuning When:

• You need the model to adopt a specific tone, style, or output format consistently.
• The knowledge is stable and unlikely to change for months.
• Latency is extremely tight and you cannot afford the retrieval overhead.
• The task is more about how the model responds than what it knows.

Use Both When:

• You need domain-specific reasoning patterns and up-to-date factual knowledge.
• Fine-tune for output structure and tone; use RAG for factual grounding.

In the system I built, the base model is not fine-tuned. All domain knowledge comes through RAG retrieval, and the model's behaviour is shaped entirely through system prompts and the context assembler's formatting. This keeps the system modular: I can swap the LLM provider without retraining, and I can update the knowledge base without touching the model. For most enterprise use cases, this is the right default. Fine-tuning becomes worthwhile when you need deeply specialised reasoning that prompt engineering alone cannot achieve - for example, generating domain-specific code or classifying documents according to a proprietary taxonomy.

Cost Comparison

RAG has higher per-query cost (embedding + retrieval + larger prompt) but near-zero update cost. Fine-tuning has lower per-query cost but significant update cost every time the training data changes. For a knowledge base that updates weekly with 500+ documents, RAG is typically 3-5× cheaper over a six-month window once you factor in fine-tuning job costs and the engineering time to manage training pipelines.

Frequently Asked Questions

What is RAG architecture?

RAG - retrieval-augmented generation - is an architecture that retrieves relevant passages from your own corpus and gives them to a language model as grounding context, so the answer is based on your documents rather than on what the model memorised in training. In production it is never a single algorithm but a system of interacting components: ingestion, chunking, embedding, indexing, retrieval, reranking, context assembly, generation and citation verification. Most RAG failures are traceable to one weak component rather than to the concept.

Why does pure vector search fail in production?

Because embeddings capture meaning, not exact strings. Vector search reliably finds passages that are semantically similar but routinely misses the exact identifiers enterprise queries depend on - policy numbers, statute references, part codes, error codes and proper nouns. Hybrid search fixes this by running dense vector search alongside sparse keyword search and fusing the two result sets before reranking. It is the single highest-value upgrade for most underperforming RAG systems.

How do you improve RAG retrieval accuracy?

Measure retrieval separately from generation first, because teams routinely blame the model for what is a retrieval failure. From there the reliable gains are hybrid search over pure vector search, a reranking pass over the initial candidate set, chunking that respects document structure rather than fixed character counts, and metadata filtering to narrow the search space before ranking. Add a citation verification step so an answer that cannot be grounded in a retrieved passage is blocked rather than returned.

What chunk size should I use for RAG?

Chunk on structure, not on a character count. A fixed 512-token window cuts through tables, splits clauses from their headings and separates definitions from the terms they define, which destroys precisely the context retrieval depends on. Chunk at the natural boundaries the document already has - sections, clauses, list items - keep the parent heading with the chunk, and overlap enough that a passage split across a boundary is still recoverable. Structure-aware chunking outperforms tuning a fixed size.

Which RAG architecture pattern should I choose?

This article describes the anatomy of one well-built pipeline, which is the right starting point for almost every team. Choosing between higher-level patterns - Hybrid RAG, GraphRAG, Agentic RAG, Corrective RAG and Multimodal RAG - is a separate decision that depends on whether your questions span relationships between documents, whether a wrong answer is expensive, and whether the sequence of steps varies by query. Start with a solid hybrid pipeline and add a pattern only when a measured failure mode demands it. I compare all five in which production RAG architecture should you build.

Building RAG That Actually Works

Production RAG is not a single algorithm - it is a system of interacting components, each of which needs to be designed, measured, and iterated on independently. The 12-component architecture I have described here is the result of months of systematic experimentation, driven by a rigorous evaluation harness and informed by real production traffic.

If you take away three things from this article, let them be these:

1. Hybrid search is non-negotiable. Vector-only retrieval leaves too much precision on the table. Adding BM25 and adaptive fusion is the single highest-leverage change you can make.
2. Chunking is architecture, not plumbing. Invest in hierarchical, structure-aware chunking with parent references and metadata enrichment. This is where most of the accuracy gains hide.
3. Measure everything. Without a labelled evaluation set and automated regression sweeps, you are guessing. Accuracy is not a single breakthrough but dozens of small, measured changes - and publishing the results, weak suites included, is what makes the number mean anything.

I am currently working on next-generation improvements including agentic retrieval (where the system autonomously decides how many retrieval passes to run and which indexes to query), and graph-augmented RAG that models entity relationships alongside vector similarity. RAG is also a critical building block in multi-agent AI systems, where retrieval agents provide grounded context to specialised reasoning agents - a pattern I explore in depth in my architecture guide. If you are building production RAG and want to compare notes, I am always happy to talk architecture. I provide RAG and AI consulting to businesses in Malaysia and Singapore.