Skip to content

Model and ingest Sources

So far, you have searched files where they sit. That is a good baseline, but the next retrieval methods need a stable way to refer to every file and every piece cut from it.

I call an original file or record a Source. The Markdown guide at docs/rollbacks.md is a Source. A row in plans.csv can be one too. When we split a Source for retrieval, each piece becomes a Chunk.

The names are plain on purpose. You will see them in test failures, Citations, cache keys, and database rows for the rest of the Course.

Give a Source a stable identity

Change into course/, then try the ID helpers:

uv run python - <<'PY'
from buildrag.ids import content_hash, source_id

path = "docs/rollbacks.md"
print(source_id(path))
print(content_hash(b"Acme Deploy rollback guide"))
PY

source_id() hashes the normalized relative path. The same path produces the same ID on macOS, Linux, and Windows. content_hash() answers a different question: did the bytes change?

I keep those identities separate because a document can change without becoming a different Source. If Acme updates docs/rollbacks.md, its Source ID stays put while its content hash changes. Citations can still point to the same Source, and ingestion knows it must rebuild the derived Chunks.

Build the Source record

Open course/src/buildrag/models.py and find the Source dataclass. It keeps the fields we will need later:

Source(
    id=source_id("docs/rollbacks.md"),
    path=PurePosixPath("docs/rollbacks.md"),
    media_type="text/markdown",
    content_hash=content_hash(raw_bytes),
    visibility=SourceVisibility.PUBLIC,
)

Do not replace this ID with an auto-incrementing database number. Row numbers tell you where a record happened to land in one database. They cannot tell two evaluation runs that they retrieved the same Source.

Make reruns boring

A useful ingestion job can run twice without duplicating anything. On each run, it should discover the current files and compare them with the stored Source records.

flowchart LR
    F[Discover files] --> H[Compute ID and hash]
    H --> C{Content changed?}
    C -->|No| K[Keep existing Chunks]
    C -->|Yes| P[Parse and rebuild Chunks]
    F --> D[Remove missing Sources]

There are three cases to handle:

  1. A new path gets a new Source record.
  2. A known path with a new content hash gets parsed and chunked again.
  3. A stored path missing from discovery is deleted with its derived Chunks.

Record the parser, chunker, and embedding versions alongside the derived data. Later, when a result changes, you will be able to tell whether the Source changed or your pipeline did.

You now have a stable object to retrieve and cite. Next, we need to turn several file formats into content we can search without flattening away the useful parts.

Parse and normalize Sources