Assemble Evidence and verified Citations

Retrieval returns Search Hits. I do not pass that list straight to the model. It may contain duplicate Chunks, five passages from one Source, or more text than the generation budget can hold.

The Evidence step turns ranked results into a smaller, labeled set that the model can use and cite.

Select a bounded set

select_evidence() deduplicates Chunk IDs, limits how many Chunks come from one Source, and stops when the text budget is full:

def select_evidence(
    hits: list[dict],
    *,
    max_characters: int,
    per_source_limit: int = 2,
) -> list[dict]:
    selected: list[dict] = []
    seen_chunks: set[str] = set()
    source_counts: dict[str, int] = {}
    used_characters = 0

    for hit in hits:
        if hit["chunk_id"] in seen_chunks:
            continue
        if source_counts.get(hit["source_id"], 0) >= per_source_limit:
            continue
        if used_characters + len(hit["text"]) > max_characters:
            continue

        selected.append({"id": f"E{len(selected) + 1}", **hit})
        seen_chunks.add(hit["chunk_id"])
        source_counts[hit["source_id"]] = (
            source_counts.get(hit["source_id"], 0) + 1
        )
        used_characters += len(hit["text"])

    return selected

The function labels the selected items E1, E2, and so on. Character count is an inspectable teaching baseline. When you know the generation model, replace it with that model's tokenizer and reserve space for instructions and output.

Source diversity is not always the right policy, but it prevents one highly repetitive guide from consuming the whole context by default.

We have chosen what the model may read. Now we need to format that Evidence so the model can tell our instructions apart from text copied out of a Source.

Format Sources as untrusted content

Give the model the Query and clearly delimited Evidence blocks:

Evidence E1
Source: docs/troubleshooting.md
Lines: 13-17
Content:
<untrusted-source>
...
</untrusted-source>

Tell the model to treat content inside the delimiter as data, not instructions. This helps with accidental instruction-following, but it is not a security boundary. We will enforce tool limits, visibility, and Citation checks in application code later.

The Evidence labels also solve another problem. Instead of asking the model to invent a path or line number, we can ask it to return the labels it used and let our code do the mapping.

Let the model cite IDs, then verify them

Ask for Evidence IDs rather than URLs or line numbers. After generation, verify those IDs and map them back to real Sources:

def verify_citations(
    cited_ids: list[str],
    evidence: list[dict],
    sources_by_id: dict[str, dict],
) -> list[dict]:
    available = {item["id"]: item for item in evidence}
    citations: list[dict] = []

    for evidence_id in dict.fromkeys(cited_ids):
        if evidence_id not in available:
            raise ValueError(f"unknown Evidence ID: {evidence_id}")

        item = available[evidence_id]
        source = sources_by_id[item["source_id"]]
        citations.append({
            "source_path": source["path"],
            "chunk_id": item["chunk_id"],
            "start_line": item["start_line"],
            "end_line": item["end_line"],
            "heading": item.get("heading"),
        })

    return citations

If the model cites E9 and you supplied only E1 through E3, the function raises an error. Valid IDs are mapped back to the Source path, heading, line range, and Chunk ID by application code. The model never gets to invent those locations.

You now have the contract generation needs: a Query, bounded Evidence, and a safe route from cited IDs to real Sources.

Generate grounded Answers