Serve RAG with FastAPI¶
I keep the API as the thinnest part of the system. An API is the interface another application uses to send a Query and receive an Answer. The retrieval decisions still belong in the pipeline you already tested.
I use FastAPI here because it is a small Python library for creating that web interface. The lesson is about exposing the RAG pipeline, not teaching general web development.
Before writing the URL handler, decide what a caller receives when the system answers, refuses, or fails. That response contract keeps failure from becoming an empty string or a vague server error.
Design the contract around failure¶
A response needs more than an Answer string. The caller should know whether the system answered, abstained, or failed during generation. It also needs verified Citations and a request ID for debugging.
from fastapi import FastAPI
from pydantic import BaseModel, Field
app = FastAPI()
class QueryRequest(BaseModel):
query: str = Field(min_length=1, max_length=4000)
debug: bool = False
class QueryResponse(BaseModel):
request_id: str
status: str
answer: str | None
citations: list[dict]
timing_ms: dict[str, float]
diagnostics: dict | None = None
@app.post("/query", response_model=QueryResponse)
def query_rag(request: QueryRequest) -> QueryResponse:
return run_pipeline(request)
Pydantic's BaseModel checks that incoming and outgoing fields have the expected types. The query_rag() function handles requests sent to the /query URL, calls run_pipeline(), and FastAPI converts the returned Python object into JSON.
Use the same run_pipeline() function in tests and evaluation. Do not create a second retrieval implementation inside the web function.
The response shape is now clear, but not every internal detail belongs in a public response. Separate what the caller needs from what only an operator should inspect.
Keep debugging details controlled¶
Normal responses can include stage timings and public Citations. Raw Search Hits, prompts, and Source text should only be visible to a user allowed to debug the system, if you expose them at all. Never return hidden model reasoning.
Set a maximum waiting time for database and model calls. Use a PostgreSQL connection pool, which keeps a small group of database connections ready for reuse. Retry only temporary failures, and keep the retry count small. Return an explicit error status when the request has used all of its allowed time or attempts.
Once the normal request and response work, you may be tempted to stream the Answer word by word. I leave that until later because it changes how errors and Citations reach the caller.
Add streaming later¶
Streaming means sending pieces of the Answer while the model is still writing. It makes the first words appear sooner, but it complicates JSON output, errors after some text was sent, and final Citations. Build the normal response first. Add streaming only after you know how the caller will receive the final status and verified Citations.
This lesson assumes the surrounding application already checks who the user is. Pass that user's access information into retrieval so visibility filters still run before ranking.
The API is not ready for real users because it returned a successful status once. Next, we will test the RAG behavior that ordinary API tests tend to miss.