Conversational RAG¶
A retriever does not understand a conversation unless you put that conversation into the retrieval step.
Suppose you are asking about single sign-on, usually shortened to SSO. It is a feature that lets someone use one login across connected services.
You ask:
What is SSO?
Then you follow with:
Which plans include it?
The second message is clear to you because you remember that it means SSO. Sent alone to lexical or vector search, it is nearly useless.
Separate the user message from the retrieval Query¶
Use the relevant conversation turns to rewrite the follow-up as a standalone Query:
Which Acme Deploy plans include SSO?
Keep both forms. The original message is what the user said and what the interface should display. The standalone Query is a retrieval aid.
I would ask the rewriting model for one field rather than a conversational response:
{"standalone_query":"Which Acme Deploy plans include SSO?"}
Validate the shape, cap the length, and fall back to the original message when rewriting fails.
Rewriting can damage intent¶
A model may resolve it incorrectly or add a product, date range, or user assumption that never appeared in the conversation. That is why Query rewriting needs its own evaluation cases.
For each conversational case, keep either the expected standalone Query or the expected Sources. Compare retrieval with and without rewriting, and inspect any case where the retrieved Evidence changes.
If the reference is genuinely ambiguous, asking the user a short clarifying question is safer than choosing one interpretation silently.
Do not carry old Evidence forever¶
Conversation history and retrieved Evidence have different lifetimes. Keep the turns needed to resolve references, but retrieve fresh Evidence for every standalone Query.
Old Chunks may be stale, and the user's access scope may have changed. Appending every previous Search Hit to history also fills the context window with content that may no longer matter.
A compact conversation summary can help with long threads, but version and evaluate the summarizer because it becomes another Query transformation.
Conversational rewriting sits before the same retrieval pipeline you already built. The next Advanced lesson looks at pushing more filtering, lexical ranking, and vector ranking into explicit SQL.