Parse and normalize Sources

Our example Source Collection contains Markdown, Python, YAML, JSON, CSV, and a deployment log. Treating all of that as anonymous prose would make retrieval worse.

I want you to normalize only what helps the formats work together. Keep the structure that makes each Source useful.

To decide what to keep, let us look at what each file format is already telling us.

Read a Source without losing its shape

For Markdown, preserve headings, code fences, and line numbers. A heading tells you what a paragraph belongs to, and the line range gives you a Citation.

For code, keep the path and line numbers. YAML and JSON are text formats that store named fields, so retain their keys and nesting. In logs, timestamps, severity, and deployment IDs are often the best search terms you have.

CSV stores tables as rows and columns in a text file. It deserves different treatment. The question How many build minutes come with Pro? names one plan and one field. Turning that row into a paragraph and hoping similarity search finds it throws away a perfectly good table. We will query the record directly in the structured retrieval lesson.

Normalize conservatively

You can safely make line endings consistent, remove known repeated boilerplate, and decode the Source with an explicit character encoding. Character encoding is the rule used to turn stored bytes into readable text. Unicode is the standard Python uses to represent those characters consistently.

Do not rewrite spelling, summarize paragraphs, or ask a model to "clean up" the text during ingestion. Those changes are hard to audit and can alter the facts.

Keep either the raw Source or its content hash. If a parser produces a strange sentence later, you need a way to compare the parsed output with the original bytes.

Here is the rule I use: normalization can change representation, but it must not change meaning.

That rule is easier to check when the parser returns a clear record. Let us sketch the record we want before we move on to chunking.

Inspect the result

Pick docs/rollbacks.md and imagine the record you want to hand to the chunker:

{
    "source_id": "src_...",
    "path": "docs/rollbacks.md",
    "media_type": "text/markdown",
    "text": "# Rollbacks\n...",
    "line_count": 42,
    "metadata": {"visibility": "public"},
}

You should still be able to answer three questions from that object: Which Source did this come from? Where in the Source was it found? What parser produced it?

Webpages and PDFs need extra machinery for network failures and page layout. Scanned PDFs may also need optical character recognition, or OCR, which turns letters visible in an image into searchable text. I leave those formats in the Reference section so they do not distract us from the retrieval pipeline we are building now.

The parser has preserved the Source. The next lesson will cut long text into Chunks while keeping their original locations attached.

Chunk without losing traceability