Parse and normalize Sources

Open course/source/acme-deploy/ and look at what is actually there. You will find Markdown, Python, YAML, JSON, CSV, and a deployment log. Treating all of that as anonymous prose would make retrieval worse.

I want you to normalize only what helps the formats work together. Keep the structure that makes each Source useful.

Read a Source without losing its shape

For Markdown, preserve headings, code fences, and line numbers. A heading tells you what a paragraph belongs to, and the line range gives you a Citation.

For code, keep the path and line numbers. For YAML and JSON, retain keys and nesting. In logs, timestamps, severity, and deployment IDs are often the best search terms you have.

CSV deserves different treatment. The question How many build minutes come with Pro? names one plan and one field. Turning that row into a paragraph and hoping similarity search finds it throws away a perfectly good table. We will query the record directly in the structured retrieval lesson.

Normalize conservatively

You can safely standardize line endings and Unicode, remove known repeated boilerplate, and decode the Source with an explicit character encoding. Do not rewrite spelling, summarize paragraphs, or ask a model to "clean up" the text during ingestion. Those changes are hard to audit and can alter the facts.

Keep either the raw Source or its content hash. If a parser produces a strange sentence later, you need a way to compare the parsed output with the original bytes.

Here is the rule I use: normalization can change representation, but it must not change meaning.

Inspect the result

Pick docs/rollbacks.md and imagine the record you want to hand to the chunker:

{
    "source_id": "src_...",
    "path": "docs/rollbacks.md",
    "media_type": "text/markdown",
    "text": "# Rollbacks\n...",
    "line_count": 42,
    "metadata": {"visibility": "public"},
}

You should still be able to answer three questions from that object: Which Source did this come from? Where in the Source was it found? What parser produced it?

Webpages and PDFs need extra machinery for network failures, page layout, and OCR. I leave those in the Reference section so they do not distract us from the retrieval pipeline we are building now.

The parser has preserved the Source. The next lesson will cut long text into Chunks while keeping that provenance attached.

Chunk without losing traceability