Rerank retrieved candidates¶
Hybrid retrieval tries to get the useful Chunks into a wider candidate list. The best supporting passage may still sit below repetitive or loosely related results.
A reranker takes that candidate list and puts it in a new order. It reads the Query and each candidate together, then gives the pair a new score. This is slower than comparing vectors, so I use it on a small candidate set rather than the whole collection.
A model that reads the Query and candidate together is called a cross-encoder.
Start with a local cross-encoder¶
uv add sentence-transformers
from sentence_transformers import CrossEncoder
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, hit.chunk.text) for hit in candidates]
scores = model.predict(pairs)
reranked = sorted(
zip(scores, candidates, strict=True),
key=lambda item: float(item[0]),
reverse=True,
)
Fetch more candidates than you plan to send to generation. For example, you might fuse 40 Chunks, rerank them, and keep 8. Those are starting values, not recommendations to copy blindly.
Keep the original retrieval method, rank, and score in your recorded details. When a result moves, you should be able to see whether word search, vector search, RRF, or the reranker caused it.
Getting the code to run is the easy part. The real decision is whether this extra model improves enough Queries to justify making every request slower.
Decide whether the extra stage earns its place¶
Reranking adds model loading time, network waiting time when the model is hosted elsewhere, and another model version that can change results. Compare the hybrid baseline with the reranked run on the 40 Evaluation Queries.
I would keep the reranker only if it puts expected Chunks higher or improves Answer quality enough to justify that cost. If expected Chunks never enter the candidate set, reranking cannot recover them. Fix retrieval first.
You should also inspect long Chunks, code, and tables. A reranker trained on short passages may score those formats poorly even when they contain the right Evidence.
Once you have the final ranked list, the next job is not generation yet. We need to choose a bounded set of Evidence, preserve Source diversity, and give every piece an ID the model can cite.