A consulting-grade reference for production RAG pipelines: nine stages from ingestion to evaluation, anchored on OpenSearch hybrid search, with orchestration framework trade-offs and a production readiness checklist.

RAG Pipeline: End-to-End Architecture Guide for Production Systems

The demo works on ten documents. At a hundred thousand, the index goes stale, someone sees a document they should not, and nobody notices retrieval got worse. The gap between demo and production is a pipeline: nine stages, each a place where a separate decision lives, each independently testable and replaceable.

We anchor the whole thing on OpenSearch because one cluster handles BM25, k-NN vectors, ACL filtering, and aggregations. Running a separate vector store means dual writes on every document update, an ACL filter that has to be kept identical in two query languages, and two failure domains to page on. Every one of those has produced a leak or a stale-index incident in engagements we have been called into.

Why Pipelines, Not Notebooks

Five failure modes recur, and each traces back to a missing pipeline stage:

  1. No reindexing strategy. Documents change. If the pipeline cannot detect updates and propagate them, every answer eventually drifts.
  2. Permission leaks. ACLs were never captured at ingest, so retrieval surfaces documents the requesting user should not see.
  3. Pure-vector recall gaps. Dense embeddings miss exact entity names, IDs, and acronyms, which is the one thing the user actually typed.
  4. No eval gates. A chunking change ships, retrieval quality drops, and nobody notices until the support queue spikes.
  5. Unobservable retrieval. Latency is logged. Recall is not.

Each stage below names which of these it prevents.

Source → Ingestion → Parsing → Chunking → Embedding → Indexing (OpenSearch)
                                                             │
                                                             ▼
  User Query → Hybrid Retrieval (BM25 + k-NN) → Rerank → Generate → Response
                                                             │
                                                             ▼
                                Eval / Observability / Online Sampling

Stage 1: Ingestion and Connectors

Prevents failures 1 and 2.

Ingestion is where documents enter the system and where ACLs are captured. Access control is an ingestion problem, not an indexing one. Source permissions (Active Directory groups, Confluence space ACLs, SharePoint sharing rules, S3 prefix policies) must become first-class metadata fields the moment a document arrives. Retrofitting them later means re-crawling every source and re-indexing everything, and the gaps in between are where leaks live.

Connector patterns split into pull (scheduled crawls) and push (webhooks, change-data-capture). Pull is simpler. Push is what you want for any source that supports it, because it gives you incremental near-real-time updates and tombstones for deletes, which is what makes reindexing (failure 1) tractable. Assume at-least-once delivery and design the downstream stages to deduplicate; without that, a retried webhook produces duplicate chunks that compete with each other in retrieval.

Managed alternative: AWS Bedrock Knowledge Bases handles connector orchestration, parsing, and indexing with less operational ownership, at the cost of less control over chunking and reranking specifics.

Stage 2: Parsing and Normalization

Feeds failure 3: a badly parsed table or a merged multi-column page never yields a chunk that matches the user's query, no matter how good the embedder is.

Parsing is its own stage because chunking quality is bounded by parse quality. PDFs with multi-column layouts, embedded tables, scanned images, and Office documents need specialised tooling; a generic text extractor will silently merge column text or skip table rows.

Tools we reach for:

  • Unstructured.io: element-level extraction (Title, NarrativeText, Table, Image), open-core, broad format support.
  • LlamaParse: cloud-hosted, strong on complex PDFs, returns markdown.
  • Azure Document Intelligence: enterprise OCR plus layout, fits Azure-heavy stacks.
  • Docling: started at IBM Research, now an LF AI & Data project, document-structure-aware, particularly strong on scientific papers.

Normalisation strips boilerplate (headers, footers, navigation), extracts title, date, author, and section hierarchy as structured metadata, and decides what to do with non-text. Tables usually convert to markdown for LLM-friendly prompts. Images either get captioned by a VLM or skipped, depending on whether the corpus is figure-heavy.

Stage 3: Chunking

The stage most often changed without an eval gate (failure 4).

Chunk size is one of the most consequential knobs in the pipeline. Too small and context is lost; too large and irrelevant tokens dilute the signal. 256-512 tokens covers most cases, with 10-20% overlap to avoid losing information at boundaries. Code, legal, and clinical text often warrant larger chunks. Tune it on your golden set.

Beyond fixed and recursive splitting, two techniques from 2024-2025 measurably improve recall:

  • Anthropic Contextual Retrieval (Sept 2024) prepends a one-to-two sentence context summary to each chunk before embedding and BM25 indexing. Anthropic's evaluation shows a 49% cut in top-20 retrieval failure rate when applied to both the embedding and BM25 indexes (5.7% to 2.9%; contextual embeddings alone: 35%, to 3.7%). Adding a reranker pushes the reduction to 67% (1.9%).
  • Late chunking embeds the full document with a long-context encoder, then segments after embedding, preserving cross-chunk context that local-window encoders lose.

Layout-aware chunking, using parse-stage structure (headings, tables, lists) as chunk boundaries, tends to beat pure token-window splitting on documents with strong structure.

Stage 4: Embeddings

Half of the answer to failure 3; the other half is Stage 6.

The embedding stage decides what "similar" means in your retrieval space. Three families:

  • Dense bi-encoders (OpenAI text-embedding-3-large, Cohere Embed 4, open-source e5/BGE) encode to a single vector, scored by cosine or dot product. The default for most workloads.
  • Sparse / learned-sparse (SPLADE and friends) keep keyword precision while gaining some semantic flexibility. A complement to dense, not a replacement.
  • Multi-vector / late-interaction (ColBERT family) store per-token embeddings and score with MaxSim. Higher recall on hard queries, larger index, more expensive to serve.

Two practical levers. Matryoshka representations allow truncating embeddings (3072 to 256 dims on text-embedding-3-large, for example) with controlled recall loss: use the truncated dim for fast first-pass retrieval and the full dim for rescoring, which is a real storage saving at 10M+ documents. Domain fine-tuning of the embedder pays off when off-the-shelf models score poorly on your golden set; synthetic query generation from your corpus is a reliable starting point, and you should budget the training and serving cost up front. The MTEB leaderboard tells you where to start, and your own golden set tells you where to stop.

Stage 5: Indexing on OpenSearch

Prevents failures 1 and 2 at the storage layer.

OpenSearch is our default because one cluster handles BM25 lexical search, k-NN vector search, ACL filtering, and aggregations. The core configuration knobs:

  • k-NN with HNSW: tune ef_construction, M, and ef_search for the recall/latency point that matches your SLA. Higher M raises recall and memory; higher ef_search raises recall and query latency.
  • Quantisation for cost. OpenSearch has offered binary quantization (1-bit, 32x compression, with oversample_factor rescoring) since 2.17. OpenSearch 3.6 added 1-bit scalar quantization on both Faiss and Lucene; Faiss 1-bit SQ is now the default for 32x compression, with 24% better recall and 15% lower latency than the earlier binary path, and Lucene gets 32x for the first time. The standard pattern is quantised retrieval for the candidate set, full-precision rescoring on the top-K.
  • ACL enforcement. Encode access groups as keyword fields and pre-filter at query time. Pre-filtering cannot leak; post-filtering can return fewer than k results and, worse, can be wired so the filter is applied after the LLM has already seen a chunk. OpenSearch's segment-level filter implementation makes pre-filtering fast enough for most workloads. Never post-filter ACLs.

Reindexing needs a strategy before go-live, because it is the mechanism behind failure 1: alias-based blue-green swaps for zero-downtime reindexing, and aliases as the routing layer for A/B testing chunking or embedding changes against live traffic.

Stage 6: Hybrid Retrieval

The direct fix for failure 3.

Pure dense retrieval consistently misses what BM25 catches: exact entity names, error codes, identifiers, acronyms. Hybrid is the default in every production system we ship, because those literal tokens are what users type when they know what they are looking for.

OpenSearch supports two fusion modes through search pipelines:

  • The normalization processor (introduced in 2.10) normalises BM25 and vector scores (min-max, L2, or z-score) and combines them with a weighted arithmetic, geometric, or harmonic mean.
  • The score ranker processor (introduced in 2.19) supports Reciprocal Rank Fusion (RRF): rank-based, parameter-free, and resistant to score-distribution drift.

Reach for RRF when you do not want to maintain calibration; reach for normalised score fusion when you need weight-tunable control per query type.

Three retrieval-side techniques worth wiring in: query rewriting with an LLM to decompose multi-part questions and expand acronyms (cheap, and it pays for itself quickly); HyDE, generating a pseudo-answer and retrieving with its embedding, which helps on ambiguous or under-specified queries; and metadata filters on date, document type, or department combined with the ACL filter, applied before scoring.

Stage 7: Reranking

Hybrid retrieval gets the right document into the top-50. Reranking moves it into the top-5, which is what the LLM reads. Cross-encoders score query and candidate jointly with full attention, far more accurately than the bi-encoder similarity of first-stage retrieval, but too expensively to apply to every candidate. The pattern: retrieve top-50 to top-100, rerank, pass top-3 to top-5 to the generator.

Options as of mid-2026: Cohere Rerank 4 (rerank-v4.0-pro / rerank-v4.0-fast, 32k context), hosted and multilingual; rerank-v3.5 is still served and is the version Amazon Bedrock exposes alongside Amazon Rerank; and BGE Reranker v2 (bge-reranker-v2-m3, bge-reranker-v2-gemma), open weights and self-hostable. In our engagements the lift is consistently worth a few hundred milliseconds of added latency. For when it pays off and how to evaluate it, see our reranking guide.

Stage 8: Generation

Generation is where the LLM finally sees the retrieved context. Three patterns determine answer quality more than the choice of model.

Context packing. Put the most relevant chunks first to mitigate the "lost in the middle" effect documented in long-context LLMs. Compress where you can: extract key sentences, summarise long passages. Leave token budget for the system prompt and the output.

Citation enforcement. Prompt the model to emit inline citations ([1], [2]) keyed to the retrieved chunk IDs, then post-process to verify cited spans exist in the retrieved context. For programmatic consumers, return structured JSON with an answer and citations array; it makes downstream verification trivial.

Guardrails. Input guardrails for topic gating and prompt-injection detection; output guardrails for hallucination, PII redaction, and brand safety. Treat hallucination detection as a separate evaluator on top of generation, not something the generator polices itself.

Stage 9: Evaluation and Operations

Prevents failures 4 and 5.

Without evaluation gates, every change to chunking, embedding, retrieval, or prompts is a coin flip. Three layers:

  • Offline regression on a curated golden set (50-500 query-context-answer triples). Metrics: context recall, context precision, answer faithfulness, answer relevancy; the RAGAS framework is the most widely adopted framing. Run on every PR.
  • Online evaluation by sampling production traffic and scoring with an LLM-as-judge. Watch for retrieval drift (recall degrading week over week) and embedding distribution shift. Calibrate the judge against human-labelled samples; we have written about where LLM judges fail.
  • CI gates. Block deployments that regress retrieval recall or generation faithfulness beyond a threshold. Promote new failure modes from production into the regression set.

Observability requires end-to-end traces: query, retrieval candidates, fused scores, reranked candidates, generation prompt, response, latency per stage, token counts. LangSmith, Langfuse, and Arize Phoenix all do this competently. Alert on latency spikes, cost anomalies, and low-confidence generations.

Orchestration: Decision Matrix

Framework Strength Pick when
LangGraph Stateful graphs, cycles, human-in-the-loop, streaming Agentic RAG with conditional branching, multi-step reasoning
LlamaIndex Workflows Tight integration with LlamaIndex connectors and indexes Already using LlamaIndex for ingestion or indexing
Haystack 2.x Pipelines Strongly-typed components, YAML serialisation, built-in evaluation Declarative pipelines and a shared component library
Plain Python Zero overhead, full control Simple linear pipelines, strong engineering discipline

Our default is plain Python. Graduate to a framework when you need branching, retries, streaming, or stateful conversations; until then plain Python is faster to debug, and a framework adopted too early is one more thing to trace through at 3am.

Cost Modelling

An illustrative per-query breakdown for a hosted-API stack. These are order-of-magnitude figures from our own engagements, not benchmarks; your numbers will move with concurrency, model choice, and caching.

Component Per-query cost (illustrative)
Query embedding ~$0.0001
OpenSearch retrieval (infra-amortised) $0.0005 - $0.002
Rerank (top-50) $0.001 - $0.003
Generation (3-5K context, frontier-class model) $0.01 - $0.05
Total $0.012 - $0.055

The biggest levers, in order of typical impact: prompt and KV caching at the LLM provider, batch embedding at ingestion, semantic caching for repeated queries (embed the query, nearest-neighbour lookup in a cache index, serve the cached answer above a similarity threshold), and reranker model choice.

Production Readiness Checklist

Before traffic switches over, verify:

  • Security and access control. ACLs verified end-to-end: ingest, index, query, response. Prompt-injection mitigations in place. Output guardrails for PII.
  • Reliability. Blue-green reindexing with rollback. Circuit breakers on LLM and reranker calls. Graceful degradation: serve without reranking if the reranker is down.
  • Quality. Golden set with at least 50 representative queries. CI eval gate that blocks on regression. LLM-judge sampling on at least 1% of production traffic.
  • Observability. Trace every request end-to-end. Dashboard with p50 and p95 latency, sampled retrieval recall, cost per query, and error rate. Alert thresholds on each.

Stages exist so that when something breaks you can point at one place, replace one component, and ship the fix without retesting the world.

Key takeaways

  • Capture ACLs at ingestion and pre-filter on them at query time. Never post-filter.
  • Hybrid retrieval is the default; pick RRF unless you have a reason to tune fusion weights per query type.
  • Quantise the candidate retrieval, rescore the top-K at full precision.
  • No chunking or embedding change ships without a golden-set eval gate.

If you are designing or hardening a production RAG system, our generative AI development services team has shipped these pipelines on OpenSearch, Elasticsearch, and Bedrock across regulated and high-throughput workloads.