Learn when cross-encoder reranking helps RAG, how to evaluate candidate coverage and ranking quality, and how to configure a documented OpenSearch rerank pipeline.

RAG Reranking: Improving Retrieval Quality with Cross-Encoders

Suppose the passage that answers a question appears at rank 17, while the application sends only the first five results to its language model. A reranker can help by rescoring the retrieved candidates. It is worth testing when the evidence is already in the candidate set; it cannot recover a passage that retrieval missed.

The idea is old by ML standards. Nogueira and Cho showed in 2019 that reranking BM25 candidates with BERT improved MRR@10 on MS MARCO by 27% relative over the previous state of the art, and the BEIR benchmark later found that reranking and late-interaction models achieve the best zero-shot results on average across 18 datasets, at a high compute cost. The engineering question is how to pay that cost only where it matters, and how to know whether it paid off.

Why first-stage retrieval is not enough

A bi-encoder represents queries and documents separately, allowing document vectors to be indexed in advance. This makes large candidate searches practical, but the ranking may miss distinctions that require comparing a query and passage together. The number of candidates and the final context size are application choices, not fixed properties of RAG.

Several issues can leave useful evidence below the final context cutoff:

  1. Chunk boundary artifacts. The sentence that answers the query is in chunk A, the context that makes it answerable is in chunk B. Neither scores high alone.
  2. Embedding collisions. Negations, conditionals and near-paraphrases collapse into nearby points. "Refunds are issued within 14 days" and "refunds are not issued after 14 days" can sit close enough to swap places under a top-k cut.
  3. Lexical gaps. Entity names, IDs, error codes and SKUs are the tokens dense embeddings handle worst and users care about most. Sparse retrieval can help both candidate coverage and ordering for exact terms.

Measure evidence coverage in the candidate set and ranking quality at the final context depth separately. A reranker may improve the second, but poor chunk boundaries or missing exact-term matches can require changes upstream.

Cross-encoders vs bi-encoders

A typical cross-encoder processes the query and passage jointly to produce a relevance score. Joint token interactions can help distinguish conditions and negations, but do not guarantee that the model understands them correctly. Unlike a bi-encoder, it usually requires query-time inference for each candidate pair, which can be batched.

That query-time work is why cross-encoders commonly follow a cheaper retrieval stage. The candidate depths below are illustrative; choose them by measuring quality and latency:

query
    ↓
  [ stage 1 ]  BM25 + kNN hybrid, ANN over the full corpus
    ↓ top 50-100 candidates
  [ stage 2 ]  cross-encoder rerank, full attention per pair
    ↓ top 3-5 passages
  LLM context

Stage 1 is tuned for recall, stage 2 for precision, and the expensive model only ever sees a hundred pairs.

Late-interaction models offer another design. ColBERTv2 retains token-level document representations and uses token interactions for scoring. It can support retrieval as well as reranking, so it is not simply another cross-encoder. Its index layout and serving costs differ from a single-vector retriever; benchmark the full deployment before choosing it.

Measure before you ship

Create a versioned evaluation set from real queries, including exact identifiers, difficult filters, and unanswerable questions. Judge the entire candidate pool you intend to rerank, or use a documented pooling strategy across systems. Labeling only the top 20 cannot fairly evaluate a reranker that promotes previously unjudged documents from ranks 21-100. Keep tuning and final test queries separate. If an LLM assists with labels, calibrate its judgments against human reviewers.

Then report before and after on the same candidate set, so the reranker is the only variable:

  • NDCG@10 as the primary, position-weighted ranking metric.
  • MRR@10 when you serve a single answer and care where the first relevant hit lands.
  • Recall@k at the depth you feed the LLM, to catch a reranker that demotes relevant passages out of the window.

Retrieval metrics are necessary, not sufficient. Pair them with one downstream metric: RAGAS gives you faithfulness and context precision, and the TREC RAG track is the closest thing to a shared methodology. Then ship behind a flag, route a slice of traffic through the reranked path, and log both ranked lists per query.

The eval set also tells you when to leave the reranker out. Reranking does not help, and can hurt, when:

  • Queries are exact-match lookups (an order ID, an error code) and BM25 already puts the answer at rank 1. A model trained on natural-language relevance can demote the exact hit in favour of a passage that reads more like an answer.
  • The reranker is off-domain or weak in the application's languages. Training corpora vary by model; test the chosen checkpoint on your own passages.
  • The latency budget is already spent. A rerank stage adds a network hop or a GPU forward pass to every query.
  • You rerank on top of hybrid search without checking the ordering: the rerank processor runs after score normalization and replaces the fused scores, so your hybrid weights no longer decide the final order.

Which reranker

Choose between an API and self-hosting based on measured quality, data handling, availability, and total operating cost. Self-hosting includes capacity planning and maintenance; hosted inference includes provider limits and possible network delays. Neither has a universally fixed cost.

A useful shortlist includes:

  • Cohere Rerank. The current catalog lists Rerank 5 Pro and Fast as its latest generation. Availability, model identifiers, limits, and billing differ by provider. Pin the model used in an evaluation and verify its API contract before upgrading.
  • BAAI BGE. The bge-reranker-v2-m3 model card provides a multilingual self-hosted candidate under Apache 2.0. Other BGE checkpoints use different base models and terms; do not infer the entire family's license from one card.
  • Jina. The v2 multilingual checkpoint is published under CC-BY-NC-4.0. Commercial deployment needs appropriate commercial terms; access to downloadable weights alone is not sufficient.
  • Sentence Transformers cross-encoders. Its pretrained model catalog includes small MS MARCO models useful as baselines. Benchmark their language and domain coverage rather than assuming a small-model result predicts every larger model.

Record truncation behavior as well as maximum context. A model with a long advertised window may be called through an integration that truncates passages earlier. Compare end-to-end latency at your candidate count and passage lengths; hardware-specific throughput tables do not establish your service's performance.

Wiring it into OpenSearch

OpenSearch can run reranking in a search pipeline, which is useful when multiple clients should share ranking behavior. Application-side reranking is also valid when you need different orchestration or failure handling. The rerank response processor can use a registered ML Commons model. The following requests follow the official Cohere integration tutorial and deliberately retain its rerank-v3.5 connector contract; this is not a claim that 3.5 is the newest Cohere model.

Before creating the connector, configure ML Commons access, connector permissions, the trusted endpoint policy, and network egress for your deployment. Use an approved secret-management process for the API key and avoid saving populated requests in source control. Register and test the model before connecting it to production search.

POST /_plugins/_ml/connectors/_create
  {
    "name": "cohere-rerank",
    "version": "1",
    "protocol": "http",
    "credential": { "cohere_key": "<your_cohere_api_key>" },
    "parameters": { "model": "rerank-v3.5" },
    "actions": [
      {
        "action_type": "predict",
        "method": "POST",
        "url": "https://api.cohere.ai/v1/rerank",
        "headers": { "Authorization": "Bearer ${credential.cohere_key}" },
        "request_body": "{ \"documents\": ${parameters.documents}, \"query\": \"${parameters.query}\", \"model\": \"${parameters.model}\", \"top_n\": ${parameters.top_n} }",
        "pre_process_function": "connector.pre_process.cohere.rerank",
        "post_process_function": "connector.post_process.cohere.rerank"
      }
    ]
  }

Register and deploy a remote model against it, and keep the returned model_id:

POST /_plugins/_ml/models/_register?deploy=true
  {
    "name": "cohere rerank model",
    "function_name": "remote",
    "connector_id": "<connector_id>"
  }

Then the pipeline, pointing at the field that holds the chunk text:

PUT /_search/pipeline/rerank_pipeline
  {
    "response_processors": [
      {
        "rerank": {
          "ml_opensearch": { "model_id": "<model_id>" },
          "context": { "document_fields": ["chunk_text"] }
        }
      }
    ]
  }

The query text the reranker compares against comes from the ext.rerank.query_context block, either as query_text or as a query_text_path into the request body (one or the other, not both):

GET chunks/_search?search_pipeline=rerank_pipeline
  {
    "size": 50,
    "query": {
      "match": { "chunk_text": "refund window for annual plans" }
    },
    "ext": {
      "rerank": {
        "query_context": {
          "query_text": "refund window for annual plans"
        }
      }
    }
  }

size is your candidate count; the processor sends every returned hit to the model, and the tutorial notes that Cohere's top_n must equal the number of documents sent. Swap the match for a hybrid query and the rerank still runs, after normalization. For a self-hosted BGE model, the same pipeline works against a SageMaker endpoint or any HTTP service that takes a query plus a list of passages, with the connector's pre- and post-process functions adapted to its payload.

Set a timeout and a documented fallback if the reranker fails. If caching scores, include the query, document content version, model version, and relevant scoring configuration in the cache key. Enforce permissions independently of that cache. Monitor error rates, latency, and labeled quality; raw score averages can shift with query mix and are not a calibrated measure of corpus drift.

Where to start

Start with the labeled set and a small reranker baseline. Compare candidate depths and models on the same queries, then evaluate final answer quality. If one model does not improve results, inspect its errors: the cause could be missing candidates, truncation, a domain mismatch, or the model itself. That result does not prove that all rerankers will fail.

For RAG on OpenSearch, keep the before-and-after candidate lists so a ranking change can be explained and rolled back.