Parse text, Markdown, and code¶
Text files look easy because Python can open them directly. I still make a few decisions before calling the result ready for retrieval.
A parser needs to choose the character encoding, preserve useful structure, and remember where every section came from.
Start with normal Python¶
For UTF-8 text, this is enough:
from pathlib import Path
path = Path("docs/rollbacks.md")
text = path.read_text(encoding="utf-8")
Character encoding is the rule used to turn stored bytes into characters. UTF-8 is the common default. If decoding fails, do not silently discard broken characters. Record the error or use a known fallback for that Source type.
def read_text(path: Path) -> tuple[str, str]:
for encoding in ("utf-8", "utf-8-sig", "windows-1252"):
try:
return path.read_text(encoding=encoding), encoding
except UnicodeDecodeError:
continue
raise ValueError(f"Could not decode {path}")
The returned encoding belongs in parser metadata so you can reproduce the result.
Preserve structure you already have¶
Markdown headings tell you where topics begin. Code fences keep examples together. In source code, paths, function names, comments, and line numbers are useful search signals.
I avoid stripping all Markdown or reformatting code before retrieval. The formatting carries meaning and makes the retrieved section easier for both the model and the reader to understand.
For a Markdown Source, I keep:
{
"source_path": "docs/rollbacks.md",
"media_type": "text/markdown",
"encoding": "utf-8",
"text": text,
"line_count": len(text.splitlines()),
}
The chunker can then split at headings while retaining line ranges.
Normalize only predictable noise¶
I normalize Windows and Unix line endings through Python's text reading. I may remove a repeated generated footer if I can identify it exactly. I do not ask a model to rewrite the Source during parsing.
Model-based cleanup makes it harder to tell whether a changed fact came from the Source or the cleanup step.
When a parsing library helps¶
A library such as Unstructured can return common element types across text, HTML, and PDF files. That can help when one ingestion system handles many formats.
It also adds dependencies and its own interpretation of structure. Compare its output with direct Python reading before adopting it for plain text.
For plain text and Markdown, native Python plus the structure-first chunker shown in the Course is enough. The code stays small enough to understand without a parsing framework.