What parsing means in RAG

When I say parsing, I mean reading a file in its original format and turning it into text or records that our code can search.

A Markdown file is already close to searchable text. A webpage contains navigation and HTML tags around the article. A PDF may store words as coordinates on a page. A CSV is a table and may be better kept as rows and columns.

The parser handles those differences.

flowchart LR
    F[Original file] --> P[Parser]
    P --> T[Searchable text]
    P --> R[Structured records]
    T --> C[Chunks]
    R --> S[Exact lookup]

What I keep from the original file

I do not want a parser to return anonymous text. I keep the Source path, file type, headings or fields, and locations such as line, page, or row numbers.

Those details help in three places:

• retrieval can use headings and fields;
• debugging can compare parsed text with the original file;
• Citations can take the reader back to the right location.

Cleaning should be conservative. Normalizing line endings is fine. Silently rewriting a sentence or summarizing a section can change the fact we meant to retrieve.

Choose by format

• Text and Markdown usually need careful reading and structure preservation.
• CSV files should often stay as records for exact lookup.
• Webpages need the main content separated from menus, footers, and other page furniture.
• PDF files may need layout extraction or OCR.

OCR means optical character recognition. It turns text visible in an image or scanned page into machine-readable characters.

I start with the simplest parser that preserves the information I need. I only add a larger parsing library after a real Source format proves the simple method insufficient.