Add ranked lexical search¶
Ripgrep gave us a trustworthy exact-search baseline. Its weakness appears when a Query matches many Chunks: every match is real, but you still need to decide which one the model should read first.
SQLite includes a full-text search engine called FTS5. We can use it without running a database server, and its BM25 ranking is easy to inspect.
Build a tiny index¶
LexicalIndex expects the Chunk objects you made in the previous lesson. Start with an in-memory database while you experiment:
import sqlite3
from buildrag.lexical import LexicalIndex
index = LexicalIndex(sqlite3.connect(":memory:"))
index.replace(chunks)
replace() clears the old rows and inserts the current Chunk IDs, Source IDs, and text. The implementation is short enough to read in course/src/buildrag/lexical.py. I recommend doing that before you run the search. FTS syntax and ranking should not feel like magic hidden behind a library method.
Now search for a phrase and an identifier:
results = index.search('"restart limit" OR DEP-1042')
for result in results:
print(f"{result.score:.4f} {result.chunk_id}")
print(result.text[:120])
The Course converts SQLite's BM25 distance so that a larger score means a better result. That convention will keep the later ranking code easier to read.
Try a Query that grep handles poorly¶
Search for deployment limit and inspect the order. Then search for undo a bad release.
BM25 rewards terms that appear in the Query, gives rarer terms more weight, and adjusts for Chunk length. It is still lexical. If the rollback guide never uses the word undo, BM25 cannot infer that undo a bad release means rollback.
That failure is useful. You now know exactly what embeddings need to improve, rather than adding them because every RAG diagram has a vector database in it.
Keep the exact method¶
Do not throw grep away. Error codes, command flags, plan names, and function names remain strong lexical targets. Later, the hybrid retriever will combine these lexical results with semantic results.
Before we add semantic search, there is another class of Query that neither grep nor embeddings should own: exact questions about structured records.