Skip to content

2024

Prompt engineering, RAG, or fine-tuning?

I see people ask whether they should use prompt engineering, RAG, or fine-tuning as if these are three versions of the same tool.

They solve different problems.

The easiest way I know to choose is to ask what the model is missing:

  • Does it need clearer instructions?
  • Does it need information it cannot currently see?
  • Does it need to repeat a behavior more consistently?

Your answer points toward prompting, RAG, or fine-tuning.

Start with prompt engineering

A prompt is the instruction and information you send to a language model for one request. Prompt engineering means writing that input so the model understands the task.

I start here because it is the fastest thing to change and test.

Prompting is useful when the model already has the information but does the task incorrectly. You might need it to:

  • return a particular JSON format;
  • follow a review checklist;
  • write in a defined tone;
  • cite only the sources you supplied;
  • say it does not know when information is missing.

A prompt cannot retrieve a private document that was never sent to the model. It also cannot hold an unlimited number of files. Every model has a limit on how much text it can read in one request.

Use RAG when information is missing

RAG stands for Retrieval-Augmented Generation. Your application searches for useful information, gives that information to the model, and then asks for an answer.

I use RAG when the answer depends on current product documentation, private company policies, source code, support records, plan details, inventory, or another live system.

RAG does not require a vector database. Exact error codes may need text search. Table values may need SQL. Questions written with different words may need vector search. The retrieval method should fit the information.

RAG also makes it possible to show Citations, which are links back to the files or records supporting the answer.

Use fine-tuning for repeated behavior

Fine-tuning means continuing to train an existing model with examples of the input and output you want.

The training changes the model's parameters. Parameters are the internal values the model learned during training. You do not edit them like database rows.

Fine-tuning can help when you have enough good examples and need a behavior to become more consistent across many requests. Examples include classification, a specialized output pattern, or a repeated transformation.

I would not use fine-tuning as the first way to add changing facts. Updating one policy or price inside a trained model is difficult to verify, and the model cannot easily show where that fact came from.

A practical way to choose

What is going wrong? I would start with Why
The output format is wrong Prompting The model needs clearer instructions.
The answer depends on today's policy RAG The model needs current outside information.
Exact error codes are missed Text search inside RAG Exact words should be searched directly.
A repeated task needs many examples in every prompt Fine-tuning evaluation Training may make the behavior more consistent.
Answers need current facts and a house style RAG plus prompting One supplies information and the other guides the writing.

You can combine them

A customer-support assistant might retrieve the current policy with RAG, use a prompt that requires Citations, and use a fine-tuned classifier to route the request to the right team.

I still test each part separately. If retrieval found the wrong policy, I fix retrieval. If the right policy was supplied but the output format is wrong, I fix the prompt. If a repeated behavior remains inconsistent across many tested examples, I consider fine-tuning.

Adding all three at once makes failures much harder to understand.

Learn how I evaluate retrieval and answers

Sources

  • in RAG
  • 3 min read

When I first wanted to build an AI chatbot with my own information, I thought I had to train the model with all of my data.

That sounds reasonable. If the model does not know your documentation, teach it your documentation.

The problem is that training is a poor fit for information that changes often. You would need good training examples, enough time and money to train, and another training run whenever the information changed. Even after that, the model could still give you a wrong answer without showing where it came from.

RAG gives us another option.

What RAG means

RAG stands for Retrieval-Augmented Generation.

It works in two simple steps:

  1. Find information related to the user's question.
  2. Give that information to a language model before it writes the answer.

A language model is the AI model that reads text and generates text. You may also see it called an LLM, which means large language model.

I like to compare RAG with an open-book exam. The model does not have to remember every fact. It can look up relevant information while answering the question.

Without retrieval, the model is taking a closed-book exam. It has to rely on what it learned during training and what you included in the question.

A small example

Imagine you have a troubleshooting guide containing this line:

DEP-1042 means the deployment restart limit was exceeded.

The user asks:

What does DEP-1042 mean?

Your application searches the guide, retrieves the matching line, and sends it to the model with the question. The model can now answer from the guide instead of guessing.

The matching line is the Evidence because it supports the answer. A link back to the guide and line number is the Citation because it shows the user where the Evidence came from.

Why people associate RAG with vector databases

Most RAG tutorials start by turning text into embeddings and storing them in a vector database.

An embedding is a list of numbers representing the meaning of text. A vector database stores those embeddings and helps you find text with similar meaning.

This is useful when the question and the document use different words. A user might ask how to undo a bad release, while the guide calls the process a rollback.

But vector search is one way to retrieve information. It is not the definition of RAG.

If the user gives you an exact error code, I would probably start with normal text search. If the answer is one value in a table, I would read that value with SQL. If the current status lives in another service, I would ask that service through its API.

All of these can be part of RAG because they retrieve information before generation.

What RAG is good at

I reach for RAG when the answer depends on information that is:

  • updated regularly;
  • private to a person or company;
  • too large to include in every request;
  • stored in files, databases, or other services;
  • expected to have a visible source.

You can update a file or database record without training the model again. You can also inspect what the system retrieved when an answer goes wrong.

What RAG does not solve automatically

RAG does not guarantee a correct answer.

The search may miss the right information. It may find the right file but rank the wrong section first. The retrieved text may contain malicious instructions. The model may ignore the Evidence or claim a Citation that does not exist.

This is why I test search and answer generation separately. If the correct text was never retrieved, changing the prompt will not fix the real problem.

RAG works best when you can see each step: what was searched, what was found, what the model received, and what supported the answer.

Start with what RAG actually means

Sources