First-stage retrieval is tuned for recall, but an LLM only reads the top 3-5 passages. A cross-encoder reranker over the top 50-100 candidates buys precision where it counts. This post covers how cross-encoders differ from bi-encoders, how to measure the lift, a verified survey of Cohere Rerank 4, BGE, Jina and ColBERT, and a working OpenSearch rerank pipeline.

RAG Reranking: Improving Retrieval Quality with Cross-Encoders

Most RAG systems I get called in to look at have the same shape of problem: the right passage is in the top 50, and the LLM never sees it because it sits at rank 17. First-stage retrieval, whether BM25, kNN or a hybrid of the two, is built to find candidates across a whole corpus. It is not built to put the best one first. A cross-encoder reranker is the piece that does that job, and it is usually the cheapest change you can make to a RAG pipeline that returns "almost right" answers.

The idea is old by ML standards. Nogueira and Cho showed in 2019 that reranking BM25 candidates with BERT improved MRR@10 on MS MARCO by 27% relative over the previous state of the art, and the BEIR benchmark later found that reranking and late-interaction models achieve the best zero-shot results on average across 18 datasets, at a high compute cost. The engineering question is how to pay that cost only where it matters, and how to know whether it paid off.

Why first-stage retrieval is not enough

A bi-encoder embeds queries and documents independently and approximates nearest neighbours in vector space. That is what makes ANN search over millions of vectors return in tens of milliseconds, and also why the right answer tends to land somewhere in the top 50 to 200 rather than at rank 1. A human scanning a result page tolerates that. An LLM reading three to five passages does not.

Three failure modes account for most of the gap between recall@100 and precision@5:

  1. Chunk boundary artifacts. The sentence that answers the query is in chunk A, the context that makes it answerable is in chunk B. Neither scores high alone.
  2. Embedding collisions. Negations, conditionals and near-paraphrases collapse into nearby points. "Refunds are issued within 14 days" and "refunds are not issued after 14 days" can sit close enough to swap places under a top-k cut.
  3. Lexical gaps. Entity names, IDs, error codes and SKUs are the tokens dense embeddings handle worst and users care about most. Sparse retrieval helps, but it fixes recall, not ordering.

One irrelevant passage in a five-passage context is enough to produce a hedge or a confident wrong answer. Recall@100 is what makes precision@5 possible; precision@5 is what the user experiences.

Cross-encoders vs bi-encoders

A cross-encoder concatenates the query and one document into a single input sequence, runs full self-attention across both, and outputs one relevance score. There is no vector to index. Every query token attends to every document token, which is why a cross-encoder catches the negation and the conditional that single-vector similarity flattens away. A bi-encoder does its document work once at index time and compares with a dot product at query time; a cross-encoder has to run a full forward pass per query-document pair, at query time, every time.

That asymmetry sets the architecture. Scoring one million documents with even a small cross-encoder at 5 ms a pair takes about 83 minutes, so a cross-encoder cannot be the first stage. The standard shape is a cascade:

query
    ↓
  [ stage 1 ]  BM25 + kNN hybrid, ANN over the full corpus
    ↓ top 50-100 candidates
  [ stage 2 ]  cross-encoder rerank, full attention per pair
    ↓ top 3-5 passages
  LLM context

Stage 1 is tuned for recall, stage 2 for precision, and the expensive model only ever sees a hundred pairs.

Late-interaction models sit between the two. ColBERTv2 (Santhanam et al., NAACL 2022) stores one small embedding per token and scores with MaxSim between query and document tokens at query time; the PLAID engine gets that to tens of milliseconds on a GPU over 140M passages. The price is index size: even with ColBERTv2's residual compression (6-10x smaller than the original ColBERT), a per-token index is far larger than a single-vector one. The reference implementation is MIT-licensed; RAGatouille wraps it.

Measure before you ship

Do this before choosing a model. Take 100 to 500 real production queries, retrieve the top 20 for each, and label every passage relevant, partially relevant or irrelevant. Domain experts give the cleanest labels; an LLM labeler with a manual spot-check scales when expert time is short. Store it as query -> [(doc_id, grade)], version it, treat it like code.

Then report before and after on the same candidate set, so the reranker is the only variable:

  • NDCG@10 as the primary, position-weighted ranking metric.
  • MRR@10 when you serve a single answer and care where the first relevant hit lands.
  • Recall@k at the depth you feed the LLM, to catch a reranker that demotes relevant passages out of the window.

Retrieval metrics are necessary, not sufficient. Pair them with one downstream metric: RAGAS gives you faithfulness and context precision, and the TREC RAG track is the closest thing to a shared methodology. Then ship behind a flag, route a slice of traffic through the reranked path, and log both ranked lists per query.

The eval set also tells you when to leave the reranker out. Reranking does not help, and can hurt, when:

  • Queries are exact-match lookups (an order ID, an error code) and BM25 already puts the answer at rank 1. A model trained on natural-language relevance can demote the exact hit in favour of a passage that reads more like an answer.
  • The reranker is off-domain. Off-the-shelf models are mostly trained on MS MARCO web passages; legal, medical and code corpora are where they regress.
  • The latency budget is already spent. A rerank stage adds a network hop or a GPU forward pass to every query.
  • You rerank on top of hybrid search without checking the ordering: the rerank processor runs after score normalization and replaces the fused scores, so your hybrid weights no longer decide the final order.

Which reranker

Two decisions: hosted or self-hosted, and how big. Hosted means zero infrastructure and a bill per search; self-hosting means data residency, fine-tuning and a flat cost, plus the deployment and monitoring work that decides build-versus-buy more often than the GPU bill does.

Cohere Rerank 4. Cohere's current models are rerank-v4.0-pro and rerank-v4.0-fast, both 32k context, 100+ languages, released December 2025; rerank-v3.5 (4k) is still served. Bedrock exposes cohere.rerank-v3-5:0 at $2.00 per 1,000 queries; on OCI, Rerank 4 is current and 3.5 is deprecated. Cohere bills in search units, one query plus up to 100 documents, with documents over 500 tokens split and each piece counted; check the Rerank 4 rate on cohere.com/pricing before doing break-even math.

BGE reranker v2 family. BAAI's bge-reranker-v2-m3 (568M parameters, Apache 2.0, built on bge-m3) is the default self-hosted starting point. bge-reranker-v2-gemma is an LLM-based reranker on gemma-2b, also Apache 2.0, and scores higher at a higher cost per pair. bge-reranker-v2.5-gemma2-lightweight is built on gemma-2-9b under the Gemma license; "lightweight" means token compression and layer cutoff at inference (BAAI reports about 60% fewer operations at compress_ratio=2), not a small model. The FlagEmbedding repo is the canonical source.

Jina Reranker v2. jina-reranker-v2-base-multilingual is a 278M-parameter cross-encoder with a 1024-token window and Flash Attention 2; Jina claims 15x the document throughput of bge-reranker-v2-m3 at similar BEIR scores. The weights are CC-BY-NC-4.0, so commercial self-hosting needs Jina's API or a marketplace package.

MiniLM baseline. cross-encoder/ms-marco-MiniLM-L6-v2 is 22M parameters, English-only, and the sbert docs list it at 74.30 NDCG@10 on TREC DL 19 at 1,800 docs/sec. Prototype with it. If it does not move NDCG@10 on your labeled set, the problem is upstream, in chunking or query understanding, and no hosted API will fix it.

Reranker Hosting Params Max tokens Languages License
Cohere Rerank 4 Pro / Fast API (Cohere, OCI) undisclosed 32k 100+ Proprietary
Cohere Rerank 3.5 API (Cohere, Bedrock) undisclosed 4k 100+ Proprietary
BGE reranker v2-m3 Self-host 568M 8k (commonly run at 512) Multilingual Apache 2.0
BGE reranker v2-gemma Self-host gemma-2b class model-dependent Multilingual Apache 2.0
BGE v2.5-gemma2-lightweight Self-host gemma-2-9b base model-dependent Multilingual Gemma
Jina Reranker v2 API or self-host 278M 1024 100+ CC-BY-NC-4.0
ColBERTv2 Self-host BERT-base class 512 per passage English-strong MIT
MS MARCO MiniLM-L6-v2 Self-host 22M 512 English Apache 2.0

Latency is left out of the table on purpose: it depends on candidate count, passage length, batch size and GPU, and the only number you should trust is the one from your own eval harness.

Wiring it into OpenSearch

The right place for the reranker is the search pipeline, not application code. The rerank response processor (ml_opensearch type, since OpenSearch 2.12) rescores the hits of any query, including hybrid ones, using a model registered in ML Commons. That keeps one ranking contract for every client, and a model swap is a pipeline update rather than a deploy. The Cohere tutorial and the SageMaker cross-encoder tutorial cover both hosted and self-hosted backends; the shape is the same.

First, a connector. The API key goes into the connector's credential block, which ML Commons encrypts at rest; there is no keystore step.

POST /_plugins/_ml/connectors/_create
  {
    "name": "cohere-rerank",
    "version": "1",
    "protocol": "http",
    "credential": { "cohere_key": "<your_cohere_api_key>" },
    "parameters": { "model": "rerank-v3.5" },
    "actions": [
      {
        "action_type": "predict",
        "method": "POST",
        "url": "https://api.cohere.ai/v1/rerank",
        "headers": { "Authorization": "Bearer ${credential.cohere_key}" },
        "request_body": "{ \"documents\": ${parameters.documents}, \"query\": \"${parameters.query}\", \"model\": \"${parameters.model}\", \"top_n\": ${parameters.top_n} }",
        "pre_process_function": "connector.pre_process.cohere.rerank",
        "post_process_function": "connector.post_process.cohere.rerank"
      }
    ]
  }

Register and deploy a remote model against it, and keep the returned model_id:

POST /_plugins/_ml/models/_register?deploy=true
  {
    "name": "cohere rerank model",
    "function_name": "remote",
    "connector_id": "<connector_id>"
  }

Then the pipeline, pointing at the field that holds the chunk text:

PUT /_search/pipeline/rerank_pipeline
  {
    "response_processors": [
      {
        "rerank": {
          "ml_opensearch": { "model_id": "<model_id>" },
          "context": { "document_fields": ["chunk_text"] }
        }
      }
    ]
  }

The query text the reranker compares against comes from the ext.rerank.query_context block, either as query_text or as a query_text_path into the request body (one or the other, not both):

GET chunks/_search?search_pipeline=rerank_pipeline
  {
    "size": 50,
    "query": {
      "match": { "chunk_text": "refund window for annual plans" }
    },
    "ext": {
      "rerank": {
        "query_context": {
          "query_text": "refund window for annual plans"
        }
      }
    }
  }

size is your candidate count; the processor sends every returned hit to the model, and the tutorial notes that Cohere's top_n must equal the number of documents sent. Swap the match for a hybrid query and the rerank still runs, after normalization. For a self-hosted BGE model, the same pipeline works against a SageMaker endpoint or any HTTP service that takes a query plus a list of passages, with the connector's pre- and post-process functions adapted to its payload.

Two things to do on day one: cache (query, doc_id) scores with a short TTL and invalidate on document update, and log first-stage scores, reranker scores and final ranks per query, alerting on a drop in mean reranker score (a corpus-drift signal) and on p99 latency.

Where to start

Build the labeled set. Run MiniLM-L6-v2 over your existing top-50 and check NDCG@10. If it moves, put the rerank processor in the pipeline with whichever model your data residency and language mix allow, and measure again on the same candidates. If it does not move, look at chunking and query rewriting before spending on a bigger model; a reranker can only reorder what stage one hands it.

If you are running RAG on OpenSearch and the answers feel almost right, reranking is usually where to look first.