Skip to content

Generate grounded Answers

We have delayed the model call until now. That was intentional. You can already inspect what was retrieved, why it ranked, and which Evidence will enter the prompt. Generation has one job left: write an Answer supported by that Evidence.

Define the response before the prompt

I use three fixed outcomes that application code can check without guessing what a sentence means:

  • answered includes text and one or more Evidence IDs;
  • insufficient_evidence includes no Answer text or Citations;
  • generation_error tells the caller that the model request failed.

The model should return one of the first two:

{"status":"answered","answer":"The deployment exceeded its restart limit.","citations":["E1","E2"]}

or:

{"status":"insufficient_evidence","answer":null,"citations":[]}

The third status comes from your application after it stops waiting, receives an invalid response, or uses all allowed retry attempts. Do not turn an API failure into a guessed Answer.

With the response contract decided, we can make the model call. I prefer deciding the output first because it gives us something concrete to validate afterward.

Make the model call small

A model provider is the company or service whose API runs the model. Different providers use slightly different request code.

I use LiteLLM, a Python library that gives those providers one common function. This keeps the lesson focused on Evidence and Answers. Install it and choose a model through configuration:

uv add litellm

from litellm import completion

response = completion(
    model=model_name,
    messages=[
        {"role": "system", "content": system_instructions},
        {"role": "user", "content": rendered_query_and_evidence},
    ],
    timeout=30,
)

Some providers can force the response into a defined JSON shape. Use that feature when available, but still check the returned JSON in your application. Validate the status, Answer type, and cited Evidence IDs. Then call verify_citations() to create the user-visible Source locations.

Gemini provides a free learning path, while GPT-5.6 Luna is the Course's inexpensive paid example. Provider limits and model names change, so keep those recommendations in the dated Reference page and record the exact model in every evaluation run.

Before testing a satisfying Answer, test the less exciting path: what happens when the Evidence cannot answer the Query?

Test the refusal path

Include unanswerable Queries in evaluation. The correct result is insufficient_evidence, not a plausible paragraph from the model's memory.

You now have a complete Query-to-Answer path. Before we let an agent choose tools or put the system behind an API, we need to measure it on more than one hand-picked example.

Evaluate the system