Chunk without losing context¶
A whole Source is usually too large and too broad to rank as one unit. A single sentence is often too small to explain anything. Chunking chooses the pieces your retriever will search.
That choice shapes the rest of the pipeline. The embedding model never sees the original document structure unless you preserve it. The Citation cannot point to useful lines unless the Chunk remembers them. Generation cannot recover a missing paragraph that the chunker separated from its heading.
Think of a Chunk as a retrieval unit¶
Suppose docs/troubleshooting.md contains one section about restart limits and another about webhook signatures. If you embed the whole file, a Query about DEP-1042 competes with every other topic in that file. If you split every sentence, the error explanation may lose the steps around it.
A useful Chunk is small enough to rank for one topic and large enough to make sense when read alone.
There is no universal size. API references, narrative policies, code, and logs have different natural boundaries. We will start with Markdown headings because the author has already told us where topics begin.
Rather than debating chunk size in the abstract, let us look at a complete structure-first implementation.
Split Markdown at headings and paragraphs¶
The code below starts a new section at each Markdown heading. If a section grows too large, it splits again at blank lines between paragraphs.
from dataclasses import dataclass
from hashlib import sha256
CHUNKER_VERSION = "markdown-sections-v1"
@dataclass(frozen=True)
class Chunk:
id: str
source_id: str
text: str
start_line: int
end_line: int
heading: str | None
def chunk_markdown(
source_id: str,
text: str,
*,
max_characters: int = 1600,
) -> list[Chunk]:
sections: list[tuple[str | None, int, list[str]]] = []
heading: str | None = None
start_line = 1
lines: list[str] = []
for line_number, line in enumerate(text.splitlines(), start=1):
is_heading = line.startswith("#") and line.lstrip("#").startswith(" ")
if is_heading:
if lines:
sections.append((heading, start_line, lines))
heading = line.lstrip("#").strip()
start_line = line_number
lines = [line]
else:
lines.append(line)
if lines:
sections.append((heading, start_line, lines))
chunks: list[Chunk] = []
for section_heading, section_start, section_lines in sections:
part: list[str] = []
part_start = section_start
for offset, line in enumerate(section_lines):
candidate = "\n".join([*part, line])
if part and len(candidate) > max_characters and not line.strip():
chunks.append(_make_chunk(
source_id, section_heading, part_start, part, len(chunks)
))
part = []
part_start = section_start + offset + 1
else:
part.append(line)
if part:
chunks.append(_make_chunk(
source_id, section_heading, part_start, part, len(chunks)
))
return chunks
def _make_chunk(
source_id: str,
heading: str | None,
start_line: int,
lines: list[str],
ordinal: int,
) -> Chunk:
body = "\n".join(lines).strip()
payload = f"{source_id}\0{CHUNKER_VERSION}\0{ordinal}\0{body}"
chunk_id = "chk_" + sha256(payload.encode()).hexdigest()[:20]
return Chunk(
id=chunk_id,
source_id=source_id,
text=body,
start_line=start_line,
end_line=start_line + len(lines) - 1,
heading=heading,
)
Each Chunk carries its parent Source ID, heading, line range, chunker version through its ID, and the text retrieval will score. Those details let you debug, rebuild, and cite the result.
This baseline uses characters because the behavior is easy to see. Models count text in tokens, which are short pieces of words and punctuation. A tokenizer is the model-specific code that splits text into those pieces. Before generation, use the tokenizer for your selected model because token count determines how much text fits and often how much the request costs.
Why overlap is not free¶
A sliding window repeats text at Chunk boundaries. That can recover a fact split across two Chunks, but it also creates near-duplicates. Repeated Chunks consume storage, crowd the top-K list, and can make the model repeat itself.
I do not add overlap by default. First, inspect failed Evaluation Queries. If relevant Evidence repeatedly straddles boundaries, add a small overlap and measure whether recall improves more than redundancy grows.
Parent-child retrieval¶
Some Queries match one precise paragraph but need the surrounding section to answer correctly. Parent-child chunking stores both:
- small child Chunks for accurate retrieval;
- larger parent sections for context expansion.
Retrieve the child, then attach its parent only when the Evidence budget allows. This works better than making every searchable Chunk large.
Version the decision¶
Changing a boundary changes Chunk IDs, embeddings, rankings, and Citations. That is why the implementation includes CHUNKER_VERSION in every Chunk ID.
After changing the chunker, compare retrieval against the same Evaluation Set. A prettier split is not automatically a better retriever.
Your Sources are now stable, parsed, and split into traceable units. We can build the first ranked index over those Chunks.