Model and ingest Sources

So far, you have searched files where they sit. That is a good baseline, but the next retrieval methods need a stable way to refer to every file and every piece cut from it.

I call an original file or record a Source. The Markdown guide at docs/rollbacks.md is a Source. A row in plans.csv can be one too. When we split a Source for retrieval, each piece becomes a Chunk.

The names are plain on purpose. You will see them in test failures, Citations, cache keys, and database rows for the rest of the Course.

Now we need to make those names useful in code. Imagine that you ingest the same folder today and again tomorrow. The system needs to recognize that docs/rollbacks.md is the same Source, even if someone edited its contents. That is why our first job is to give every Source a stable identity.

Give a Source a stable identity

We will create one identity from the file path and another fingerprint from its contents. Both helpers use SHA-256, a standard hashing algorithm:

from hashlib import sha256
from pathlib import PurePosixPath


def source_id(path: str) -> str:
    normalized = PurePosixPath(path.replace("\\", "/"))
    if normalized.is_absolute() or ".." in normalized.parts:
        raise ValueError("Source paths must remain relative")
    digest = sha256(normalized.as_posix().encode()).hexdigest()[:16]
    return f"src_{digest}"


def content_hash(content: bytes) -> str:
    return sha256(content).hexdigest()

A hash is a short digital fingerprint calculated from some input. The same input produces the same fingerprint, while changed input should produce a different one.

source_id() hashes the cleaned relative path. The same path produces the same ID on macOS, Linux, and Windows. content_hash() fingerprints the file contents and answers a different question: did the file change?

I keep those identities separate because a document can change without becoming a different Source. If Acme updates docs/rollbacks.md, its Source ID stays put while its content hash changes. Citations can still point to the same Source, and ingestion knows it must rebuild the derived Chunks.

At this point, we can identify a Source and notice when it changes. An ID and hash are not enough on their own, though. Retrieval will also need the path, file format, and access level. We will keep those details together in one Source record.

Build the Source record

A dataclass is a Python class used mainly to hold named fields. This Source record keeps the details we need:

from dataclasses import dataclass, field
from typing import Any


@dataclass(frozen=True)
class Source:
    id: str
    path: PurePosixPath
    media_type: str
    content_hash: str
    visibility: str = "public"
    metadata: dict[str, Any] = field(default_factory=dict)


raw_bytes = b"Acme Deploy rollback guide"
source = Source(
    id=source_id("docs/rollbacks.md"),
    path=PurePosixPath("docs/rollbacks.md"),
    media_type="text/markdown",
    content_hash=content_hash(raw_bytes),
)

The media_type value says what format the Source uses, such as text/markdown. visibility records who may retrieve it.

Do not replace the Source ID with an automatically assigned database number. Row numbers tell you where a record happened to land in one database. They cannot tell two evaluation runs that they retrieved the same Source.

We now know what one Source record looks like. The next question is what happens when we run ingestion again. Real files change, new files appear, and old files disappear. A good ingestion job handles all three without creating duplicates.

Make reruns boring

On each run, discover the current files and compare them with the Source records you stored last time.

flowchart LR
    F[Discover files] --> H[Compute ID and hash]
    H --> C{Content changed?}
    C -->|No| K[Keep existing Chunks]
    C -->|Yes| P[Parse and rebuild Chunks]
    F --> D[Remove missing Sources]

There are three cases to handle:

1. A new path gets a new Source record.
2. A known path with a new content hash gets parsed and chunked again.
3. A stored path missing from discovery is deleted with its derived Chunks.

Record the parser, chunker, and embedding versions alongside the derived data. Later, when a result changes, you will be able to tell whether the Source changed or your pipeline did.

You now have a stable object to retrieve and cite. Next, we need to turn several file formats into content we can search without flattening away the useful parts.

Parse and normalize Sources