Skip to content

What is RAG? It is more than vector search

When I first wanted to build an AI chatbot with my own information, I thought I had to train the model with all of my data.

That sounds reasonable. If the model does not know your documentation, teach it your documentation.

The problem is that training is a poor fit for information that changes often. You would need good training examples, enough time and money to train, and another training run whenever the information changed. Even after that, the model could still give you a wrong answer without showing where it came from.

RAG gives us another option.

What RAG means

RAG stands for Retrieval-Augmented Generation.

It works in two simple steps:

  1. Find information related to the user's question.
  2. Give that information to a language model before it writes the answer.

A language model is the AI model that reads text and generates text. You may also see it called an LLM, which means large language model.

I like to compare RAG with an open-book exam. The model does not have to remember every fact. It can look up relevant information while answering the question.

Without retrieval, the model is taking a closed-book exam. It has to rely on what it learned during training and what you included in the question.

A small example

Imagine you have a troubleshooting guide containing this line:

DEP-1042 means the deployment restart limit was exceeded.

The user asks:

What does DEP-1042 mean?

Your application searches the guide, retrieves the matching line, and sends it to the model with the question. The model can now answer from the guide instead of guessing.

The matching line is the Evidence because it supports the answer. A link back to the guide and line number is the Citation because it shows the user where the Evidence came from.

Why people associate RAG with vector databases

Most RAG tutorials start by turning text into embeddings and storing them in a vector database.

An embedding is a list of numbers representing the meaning of text. A vector database stores those embeddings and helps you find text with similar meaning.

This is useful when the question and the document use different words. A user might ask how to undo a bad release, while the guide calls the process a rollback.

But vector search is one way to retrieve information. It is not the definition of RAG.

If the user gives you an exact error code, I would probably start with normal text search. If the answer is one value in a table, I would read that value with SQL. If the current status lives in another service, I would ask that service through its API.

All of these can be part of RAG because they retrieve information before generation.

What RAG is good at

I reach for RAG when the answer depends on information that is:

  • updated regularly;
  • private to a person or company;
  • too large to include in every request;
  • stored in files, databases, or other services;
  • expected to have a visible source.

You can update a file or database record without training the model again. You can also inspect what the system retrieved when an answer goes wrong.

What RAG does not solve automatically

RAG does not guarantee a correct answer.

The search may miss the right information. It may find the right file but rank the wrong section first. The retrieved text may contain malicious instructions. The model may ignore the Evidence or claim a Citation that does not exist.

This is why I test search and answer generation separately. If the correct text was never retrieved, changing the prompt will not fix the real problem.

RAG works best when you can see each step: what was searched, what was found, what the model received, and what supported the answer.

Start with what RAG actually means

Sources