Test RAG-specific behavior¶
I can write a normal API test that proves /query returns JSON, but that tells me very little about the quality of the RAG system. It does not prove that ingestion removed a deleted Source, private files were filtered before ranking, or a Citation points to the Evidence the model received.
I test the boundaries between data, retrieval, and generation because that is where RAG failures hide.
Keep repeatable behavior fast¶
Small tests should describe the retrieval rule directly. For example:
def test_rrf_promotes_a_chunk_found_by_both_methods():
scores = reciprocal_rank_fusion([
["exact", "shared"],
["shared", "semantic"],
])
assert scores["shared"] > scores["exact"]
assert scores["shared"] > scores["semantic"]
def test_cache_key_changes_when_the_index_changes():
first = cache_key(
"retrieval", query="rollback", index_version="v1"
)
second = cache_key(
"retrieval", query="rollback", index_version="v2"
)
assert first != second
Other fast tests should cover stable Source and Chunk IDs, traceable line ranges, cosine similarity, Evidence budgets, and Citation validation.
These tests should not call a model or a hosted database. The same input should always produce the same result, which keeps the tests fast and repeatable. Run them on every code change so silent identity and ranking changes become visible.
Individual functions are only the beginning. Ingestion has behavior spread across several runs, so test the full life of a Source rather than one successful insert.
Test ingestion as a lifecycle¶
A successful insert is only one case. Run ingestion twice and confirm that unchanged Sources do not duplicate Chunks. Change a Source and confirm that its hash and derived Chunks update. Delete it and confirm that stale Chunks disappear.
Then test visibility inside the retrieval query, not as a cleanup step after results return.
The happy path now has coverage. Next, make the failures happen deliberately so you know which status and logs the application produces.
Test failures on purpose¶
Use saved fake model responses to test:
- a valid answered result;
insufficient_evidence;- an unavailable Citation ID;
- invalid JSON;
- a timeout followed by bounded retries;
- a provider failure that returns
generation_error.
The malicious prompt-injection Source and private Zenith Source are saved test inputs, not examples to check by hand once. Add them to integration tests, which run several parts of the pipeline together, along with malformed tool arguments and oversized requests.
These tests protect known rules. The Evaluation Set has a different job: detecting whether search and Answer quality became worse across many Queries.
Keep evaluation as a separate gate¶
The 40-question Evaluation Set catches cases where retrieval quality becomes worse even though individual functions still pass their tests. Run the repeatable retrieval evaluation automatically on every pushed change. Keep paid or variable model-based Answer evaluation for scheduled runs or release checks, and compare it with a saved earlier result.
A test failure should point you toward a stage. If retrieval misses the Source, do not tune the prompt. If Evidence is correct but the model invents a Citation, fix generation validation.
The last lesson packages these tested behaviors with explicit operational limits and deployment configuration.