Secure RAG and Agentic Retrieval¶
I treat every retrieved Source as untrusted. A RAG system reads content you did not write into the model's context, and an agent may choose its next tool based on that content.
One attack is called prompt injection. It happens when a Source contains text that tries to give instructions to the model instead of supplying information.
The Acme Deploy Source Collection includes security-fixtures/prompt-injection.md. Open it. The file tells the model to ignore its real instructions and reveal secrets. Your retriever is allowed to find and quote that text. The model is not allowed to obey it.
Treat Sources as data¶
Wrap each Evidence block in clear delimiters and tell the model that Source text is untrusted. This gives the model a useful boundary, but application code must enforce the real controls.
A Source cannot grant itself more visibility. It cannot enable a tool. It cannot change the number of allowed turns. It cannot create a valid Citation by writing E1 inside its own text.
Treating text as data is the first boundary. The next boundary decides which Sources may enter retrieval for the current user at all.
Filter before ranking¶
The collection also contains private/zenith-project.md. Imagine the current user can read only public docs.
The visibility condition must run before lexical scoring, vector ranking, reranking, and context assembly. Filtering after top-K can leak private information through returned text, scores, logs, or even the absence of expected public results.
The surrounding application should tell retrieval which Sources the current user may access. This Course does not teach login systems, but the RAG layer must respect the access decision it receives.
Filtering controls what the agent can see. Tool limits control what it can do and how much work one request can consume.
Bound every tool¶
For an agent, validate tool names and arguments in the harness. Restrict file paths to the Source root and limit result counts, output size, waiting time, and loop turns.
Keep user-provided values separate from SQL commands instead of joining them into a SQL string. Database libraries call this a parameterized query, and it prevents user text from becoming part of the command.
A read-only agent can still cause harm through expensive repeated calls or sensitive output. Read-only describes what it can change, not how much it can cost or reveal.
Verify the output¶
Reject Citation IDs that were not in the supplied Evidence. Return insufficient_evidence when retrieval is empty or unsupported. Do not let a model fill a generation error with a plausible guess.
Keep raw Queries, Evidence, Answers, and prompts out of normal logs. Store IDs, timings, statuses, and scores. If debugging needs the protected content, retrieve it through an authorized path rather than copying it into every trace.
These rules are easy to follow in one manual example and easy to break during a later refactor. Put the attacks into the test suite so the safe behavior stays visible.
Test the attacks as fixtures¶
Add the malicious Source, private Source, malformed tool arguments, fabricated Citation IDs, oversized Queries, and repeated tool loops to automated tests. Security behavior should fail in CI when the pipeline changes.
Once these boundaries are in place, we can optimize expensive stages without accidentally caching private or stale results.