Search and read Sources with ripgrep

FoundationsContent SearchFirst evaluation

Glob gave you a list of possible Sources. Ripgrep, usually called through the rg command, opens those files and finds matching text.

This search-then-read pattern is one of the most useful ideas in retrieval. Search tells you where a possible answer lives. Reading a small window around the match gives you enough context to decide whether it is Evidence.

Why agents do not read the whole file

A search match is often one line. One line may be too little to understand. The full file may be hundreds or thousands of lines, which is too much to put into model context every time.

I split the work into two tools:

1. search returns paths, line numbers, and matching lines;
2. read returns a small line range around a promising match.

This saves space and keeps a record of where the text came from. If the Answer uses the passage, you already know the Source and line range for its Citation.

Find an exact identifier

For the example files, an exact search for the fictional deployment error looks like this:

rg --line-number --fixed-strings "DEP-1042" acme-deploy

--fixed-strings tells ripgrep to search for those exact characters. Without it, ripgrep can treat the input as a regular expression, which is a small pattern language for matching text.

In retrieval, you will often see the user's question called a Query. I will use that word from now on. An exact search is a good default when the Query contains an error code, function name, command flag, or another exact term.

The result contains three useful pieces:

path:line_number:matching text

Do not strip the first two away. A text match without its location is hard to inspect and impossible to cite precisely.

You can narrow the search when the Query suggests a format:

rg --line-number --glob "*.md" "restart limit" acme-deploy

The terminal output is useful for a person. An application needs the same path, line number, and text in a predictable Python object.

Call ripgrep from Python

Ripgrep can return one JSON event per line. The function below keeps only match events and turns them into GrepMatch objects:

import json
import subprocess
from dataclasses import dataclass
from pathlib import Path, PurePosixPath


@dataclass(frozen=True)
class GrepMatch:
    source_path: PurePosixPath
    line_number: int
    line: str


def grep_sources(
    root: Path,
    pattern: str,
    *,
    literal: bool = False,
    limit: int = 100,
) -> list[GrepMatch]:
    command = ["rg", "--json", "--color", "never"]
    if literal:
        command.append("--fixed-strings")
    command.extend((pattern, "."))

    completed = subprocess.run(
        command,
        cwd=root,
        check=False,
        capture_output=True,
        text=True,
    )

    if completed.returncode not in (0, 1):
        raise RuntimeError(completed.stderr.strip() or "ripgrep failed")

    matches: list[GrepMatch] = []
    for raw_event in completed.stdout.splitlines():
        event = json.loads(raw_event)
        if event.get("type") != "match":
            continue

        data = event["data"]
        matches.append(GrepMatch(
            source_path=PurePosixPath(data["path"]["text"]),
            line_number=int(data["line_number"]),
            line=data["lines"]["text"].rstrip(),
        ))

        if len(matches) >= limit:
            break

    return matches

A separate read function can return only the nearby lines:

def read_source_window(
    root: Path,
    source_path: PurePosixPath,
    line_number: int,
    *,
    context: int = 2,
) -> str:
    resolved_root = root.resolve()
    path = (root / source_path).resolve()
    if not path.is_relative_to(resolved_root) or not path.is_file():
        raise ValueError("source path must stay inside the Source Collection")
    if line_number < 1 or context < 0:
        raise ValueError("invalid line range")

    lines = path.read_text().splitlines()
    start = max(0, line_number - context - 1)
    end = min(len(lines), line_number + context)
    return "\n".join(
        f"{index + 1}: {lines[index]}"
        for index in range(start, end)
    )

For DEP-1042, the matching Source is docs/troubleshooting.md, and the nearby text explains that the restart limit was exceeded. That nearby text is the Evidence.

Now we can compare the result with a question whose expected Source is already recorded in the Evaluation Set. This gives us a baseline before we add ranking or embeddings.

Exact search is a baseline, not a toy

The Evaluation Set includes the Query What does error DEP-1042 mean? The expected Source is docs/troubleshooting.md, and the expected Evidence contains restart limit exceeded.

Ripgrep handles this case extremely well. An embedding model would add cost and may rank the exact identifier less reliably.

A harder Query shows the limit:

rg --line-number --ignore-case "undo a bad release" acme-deploy/docs

This may return no useful match even though the rollback guide answers the question. The guide uses different words. Later, we will add semantic search, which searches by similar meaning instead of requiring the same words.

Before that, we will improve lexical search. Lexical search means searching and ranking the words that appear in the Query and Sources.

One more thing before we leave exact search: decide what the tool is allowed to search and how much text it may return. Those limits belong in the design, not as cleanup after a large result.

What can go wrong

Keep the search root inside the Source Collection. Limit result counts and bytes. Treat regular expressions from users as untrusted input, and do not return an entire large file when five nearby lines answer the Query.

I treat these limits as part of the search tool itself. The model can ask for a search, but the software around it must decide what paths, patterns, and output sizes are allowed.

You have now used the same discover, search, and bounded-read loop that coding agents use on real repositories. In the next lesson, I will show you how those tools fit around a model.

See how coding agents retrieve context