Monitor a RAG request end to end¶
When a request takes four seconds, I want to know where those four seconds went. The total time does not tell me whether PostgreSQL was slow, the reranker stopped waiting, the model tried again, or retrieval returned nothing.
I record a trace, which is a timeline of the steps inside one request. Each step records how long it took and whether it succeeded. This lets me inspect retrieval, Evidence selection, and generation separately.
Time the stages you can act on¶
A small context manager can record how many milliseconds each stage takes:
from collections.abc import Iterator
from contextlib import contextmanager
from dataclasses import dataclass, field
from time import perf_counter
@dataclass
class RequestTrace:
request_id: str
stage_ms: dict[str, float] = field(default_factory=dict)
retrieval_methods: list[str] = field(default_factory=list)
cache_hits: dict[str, bool] = field(default_factory=dict)
@contextmanager
def stage(self, name: str) -> Iterator[None]:
started = perf_counter()
try:
yield
finally:
elapsed = (perf_counter() - started) * 1000
self.stage_ms[name] = round(elapsed, 3)
trace = RequestTrace("req_01")
with trace.stage("retrieve"):
hits = retrieve(query)
with trace.stage("select_evidence"):
evidence = select(hits)
The trace stores durations and safe metadata without recording raw Query or Evidence text.
A useful request record includes:
- request and trace IDs;
- retrieval methods and configuration versions;
- returned Chunk IDs, ranks, and scores;
- cache hits and misses;
- fallback and retry events;
- Evidence IDs and Citation count;
- Answer status;
- token usage, estimated cost, and stage timings.
Those fields let you follow one bad Answer without copying private Source content into the logging system.
A trace explains what happened on a real request. It does not tell us whether the system remains accurate across the Evaluation Set, so I keep monitoring and evaluation as separate views.
Separate operation from evaluation¶
Monitoring tells you what the live system is doing: latency rose, empty retrieval doubled, or model errors appeared. Evaluation tells you whether the system still retrieves and answers correctly on labeled Queries.
You need both. A fast system can be wrong, and a high-scoring offline run can fail under real timeouts and provider limits.
The small RequestTrace works inside one Python process. If the request later crosses several services, we need a way to connect the pieces.
Add tracing when the system crosses boundaries¶
Structured JSON logs are enough for the first service. When the request crosses several programs or vendors, you can add OpenTelemetry, a shared standard for recording and connecting traces. Use the request ID to connect those traces with evaluation files and user-reported failures.
Alert on sustained retrieval emptiness, invalid Citation attempts, generation errors, latency, and cost. A single strange Query is something to inspect. A trend is something to page on.
We now have enough visibility to put a stable HTTP boundary around the pipeline without hiding its behavior.