Cache expensive RAG stages

A cache stores the result of completed work so the application can reuse it later. The same Source may be embedded more than once, and the same Query may run the same retrieval plan. A cache can skip that work, but it can also return old or private data very quickly.

I treat a cached value as valid only while every input that shaped it remains the same.

Why the Query is not enough

Suppose two users ask What is the Zenith database host? One can access the private Zenith Source and one cannot. A retrieval cache keyed only by Query text could hand the second user the first user's results.

The same problem appears when a Source changes, you switch embedding models, or you rebuild the index. The words of the Query did not change, but the correct result may have.

A safe cache therefore needs more than the Query as its label. We will build the label from every input that can change the result.

Build versioned keys

A cache key is the label used to store and find a cached value. The label must describe every input that can change the result.

The function below sorts those named inputs and hashes them into a stable key:

from hashlib import sha256


def cache_key(
    namespace: str,
    **dimensions: str | int | bool,
) -> str:
    if not namespace or not dimensions:
        raise ValueError("cache keys require a namespace and dimensions")

    payload = "\0".join([
        namespace,
        *(f"{name}={dimensions[name]}" for name in sorted(dimensions)),
    ])
    digest = sha256(payload.encode()).hexdigest()
    return f"{namespace}:{digest}"


key = cache_key(
    namespace="retrieval",
    query="how do I undo a bad release?",
    visibility="public",
    retrieval_version="hybrid-rrf-v2",
    index_version="sources-2026-08-14",
    top_k=8,
)

namespace separates retrieval entries from other types of cached data. The other named values describe the Query, access level, retrieval version, index version, and result count.

For an embedding cache, include the Source content hash, Chunk ID or text hash, embedding model revision, prefixes, and normalization settings.

For retrieval results, include the normalized Query, access scope, retrieval configuration, index version, and top-K.

I would avoid Answer caching until you have a measured reason for it. An Answer also depends on Evidence, generation model, prompt version, access scope, and policy. If you cache it, retain the original verified Citations and include all of those dependencies.

Detailed keys handle changes we can name. We still need an expiry rule for changes outside our control.

Invalidate by identity and time

Versioned keys make changed inputs miss the old cache automatically. A time-to-live sets how long an entry may be reused before it expires. This helps when an outside service changes without giving you a version number.

Track how often the cache is used, how much waiting time it saves, how much it costs to store, and whether it returns old results. If a cache rarely helps or makes failures hard to explain, remove it.

Now that some stages may disappear behind cache hits, request-level timing alone becomes even less informative. We need to trace each stage and record whether it actually ran.

Monitor the pipeline