Evaluate retrieval and Answers separately¶
A convincing Answer can hide bad retrieval. The model may know the topic already, guess correctly, or write around missing Evidence. If you score only the final prose, you will not know which part of the system improved or broke.
I separate retrieval evaluation from Answer evaluation and join them only after both are visible.
Meet the Evaluation Set¶
The example Evaluation Set contains 40 Queries in five groups:
- exact Queries contain words or identifiers that should match directly;
- semantic Queries ask with different words but the same meaning;
- structured Queries ask for values from a table or record;
- multi-source Queries need information from more than one Source;
- unanswerable Queries ask for information that is not in the files.
Each answerable record names the Sources and quotes the Evidence I expect the system to find in the pretend Acme Deploy collection. The unanswerable records let us test whether the system refuses to guess.
Before calculating any score, validate those expectations against the current Sources. If a Source was renamed or an expected sentence disappeared, report the stale evaluation record instead of counting it as a retrieval failure.
Once the questions and expected Sources are valid, score the first part of the system on its own: did retrieval find what the Answer needed?
Score retrieval first¶
I start with Recall@K. K is the number of top results you inspect, so Recall@5 asks whether the expected Source or Chunk appeared in the first five results.
Mean Reciprocal Rank, or MRR, rewards systems that put the first correct result near the top. Normalized Discounted Cumulative Gain, or nDCG, is useful when several results can be relevant and some are better than others. You do not need all three metrics on day one. Add a metric when it answers a ranking question you care about.
Keep latency and empty-result rate beside the quality metrics. A retriever that gains one point of recall by taking ten times longer may still be the wrong trade.
When a Query fails, save the returned Chunk IDs and scores. Read the actual miss. Aggregate numbers tell you that quality changed; failed examples tell you what to fix.
Only after retrieval is visible do I score the generated Answer. This stops a fluent model from hiding the fact that the right Source was never found.
Score the Answer against supplied Evidence¶
Check whether every Citation points to supplied Evidence, whether the Evidence supports the claim, and whether required claims have Citations. Track correct abstention separately from generation errors.
Use checks with repeatable results whenever you can. If you ask another language model to grade groundedness or relevance, keep the grading instructions and model version, save its explanation, and compare it with examples you reviewed yourself. A model grader can vary between runs, so I do not treat it as the only quality check.
We now have separate retrieval and Answer scores. To make them useful, each run needs a saved record that we can compare with the next change.
Compare one change at a time¶
Store each run as JSONL, a text format containing one JSON record per line. Keep the Query ID, settings, model versions, retrieved IDs, scores, Answer status, Citations, timing, cache state, and cost. Change one part of retrieval, rerun the same set, and compare it with the accepted baseline.
You have now built and measured every stage separately. The capstone connects them into one PostgreSQL-backed request path before we let a model choose retrieval tools.