Elastic: jina-reranker-v3.5 improved semantic search most on average with 20 candidates
In Elastic’s experiment across 25 benchmarks, jina-reranker-v3.5 cut semantic search error most on average with 20 candidates and BM25 error most with 100. When semantic search supplied 200 candidates, results were worse on average than without reranking.
In a post on Elastic’s Search Labs blog on 13 November 2026, Scott Martens and Sofia Vasileva tested how many search results the company’s own jina-reranker-v3.5 model should rerank. They ran the model on 25 benchmarks drawn from the BEIR, CQADupStack, CoIR and BRIGHT collections.
A reranker takes the candidates found by fast first-stage retrieval and orders them by relevance. The model tested is a listwise reranker: it reads the query and the whole candidate list in a single pass. When a model scores candidates one by one, extra candidates do not change how the others are ranked. In a listwise model, they can degrade the whole result. The authors attribute this to the model’s attention spreading and noise accumulating.
The best number of candidates depended on first-stage retrieval
The first stage was either Elasticsearch BM25 keyword search or semantic vector search with the jina-embeddings-v5-text-small model. The authors measured relevance with nDCG@10, which assesses how well the most relevant documents rank among the top ten results. The table shows the average share of error that reranking removed compared with first-stage retrieval alone (0.2097 ≈ 21 per cent).
| Candidates | BM25 | Semantic search |
|---|---|---|
| 10 | 0.0878 | 0.0661 |
| 20 | 0.1433 | 0.0848 |
| 50 | 0.1875 | 0.0767 |
| 100 | 0.2097 | 0.0629 |
| 200 | 0.1873 | −0.0239 |
Source: Elastic, average of 25 benchmarks, nDCG@10. BM25: Elasticsearch Serverless, English analyser, default settings; retrieval returned at most 200 candidates, so larger numbers were not tested.
The average hides wide variation. With semantic search, reranking lowered scores on the BRIGHT LeetCode and CoIR CoSQA benchmarks at every candidate count, although CoSQA results varied by about 0.01 between runs. With BM25, only three CoIR benchmarks gained more from 200 candidates than from smaller numbers.
The authors acknowledge that the test is not scientifically rigorous. They left out the largest BEIR datasets, and candidate counts were fixed. The results may not apply to other rerankers.
With BM25, 50 candidates raised error by about 2 per cent compared with 100 but halved the number of tokens. The authors consider 50 the most cost-effective choice.
BM25 with reranking comes close to semantic search
Semantic search alone reduced error by 25.7 per cent on average compared with BM25. When the top 100 BM25 results were reranked, semantic search reduced error by only 4.2 per cent compared with that combination. BM25 with reranking beat semantic search on half of the BEIR benchmarks.
However, semantic search with 20 reranked candidates outperformed BM25 with 100 reranked candidates on every benchmark, with error 13.26 per cent lower on average.
The model is available from Elastic, the Jina API and Hugging Face
In Elasticsearch, the model is added as a text_similarity_reranker retriever, and the number of candidates is set with rank_window_size. Elastic recommends 100 as a starting point for BM25, with 50 for comparison, and 20 for semantic search, but still urges users to measure the right number on their own data. The model is available in Elastic Inference Service, through the Jina API and as a self-hosted container. The model weights on Hugging Face are licensed under CC BY-NC 4.0 for non-commercial use; commercial use is arranged with Elastic.
Read more: Models