Prompt injection is what happens when text that reaches a language model gets read as instructions instead of data. An attacker (or an innocent document) tells the model to ignore its earlier instructions, reveal a system prompt, or call a tool it should not. The reason it keeps catching teams out is that a model has no reliable way to separate the developer's intent from any other text in its context window. People reach for one defence, usually an input filter or a classifier that scores a request as safe or unsafe, and treat the problem as solved. A single filter never holds, because the attack surface is the whole context window, and the useful instructions and the malicious ones are made of the same material. What works in production is defence in depth: several independent controls, each assuming the others can fail.
Direct and indirect prompt injection
It helps to split the problem in two. Direct injection is the obvious case: a user types something into the chat box that tries to override your instructions. You can see it, you can log it, and a lot of the public examples live here. It is the easier half to reason about because the hostile text and the user are the same party.
Indirect injection is the half that does real damage. The malicious instructions are not typed by the user at all. They are hidden inside content the model retrieves or a tool returns: a web page the agent summarises, a support ticket it triages, a PDF in a knowledge base, the body of an email it reads, the JSON a third-party API hands back. The user asks an innocent question, your retrieval step pulls in a poisoned document, and the document quietly tells the model to forward the conversation somewhere or call an internal tool. Any agent that reads untrusted data and can also act on the world has this exposure, and most useful agents read untrusted data.
Once you frame it this way the conclusion follows. You cannot trust the boundary between instruction and data inside the model, so you have to enforce trust boundaries outside it, around what the model is allowed to touch. That is the rest of this article. If you are designing the agent loop itself, the related piece on LLM orchestration covers where these controls sit in the flow of a request.
Treat model output as untrusted input
The first habit to build is to stop trusting what the model produces. The output of one model call is, for security purposes, the same as any other untrusted string. If the model has read a poisoned document, its next message can carry the attacker's intent forward. Teams that pipe model output straight into a shell command, a database query, an HTTP request, or another agent's prompt have handed the attacker a path right through their system.
The fix is the same discipline you would apply to any input from the outside: never interpolate raw model output into a privileged operation. If the model decides to run a query, it should select from a fixed set of parameterised queries and supply typed arguments, never assemble SQL as a string. If it renders text back to a browser, escape it. If one agent passes work to another, the receiving agent should treat the handoff as data to be validated, not as instructions to obey.
Restrict tools with allowlists and typed parameters
A model that can only talk is hard to weaponise. A model that can call tools is where injection turns into impact, so the tools are where most of the defence lives. The principle is to make the set of possible actions small, explicit, and strongly typed, so that even a fully hijacked model can only do things you already decided were acceptable.
Allowlist the actions, not the inputs
Give the agent a closed list of tools it may call, and refuse anything outside it. Resist the general-purpose escape hatches: an arbitrary HTTP fetch, a shell command, an "execute this code" tool. Each of those collapses your allowlist back to "anything". If the agent needs to reach an internal service, give it a specific tool for that service with the endpoint fixed in your code, not a parameter the model fills in.
Type and validate every parameter
Define tool parameters with a strict schema, validate against it before the call runs, and constrain the values. An account identifier is a known format, a date is a date, an amount has a ceiling. A typed boundary turns a vague "the model asked me to do something" into a concrete check you can reject. This is the same instinct as input validation in any API, applied to the model's requests.
Least-privilege credentials and limiting blast radius
Assume, for the sake of design, that an attacker can make the model call any tool it has access to. The question then becomes: how much damage does that allow? You want the answer to be small, and that is a question about credentials and scope, not about the model.
Give the agent its own identity with the narrowest permissions that let it do its job. If it reads from one table, it gets read on one table, not a shared service account with write access to everything. Scope tokens to the current user and the current task, so a hijacked session cannot reach another tenant's data. On our work with Darktrace the OAuth and JWT handling was built so that each request carried only the scopes it needed and expired quickly, which is exactly the property you want behind an agent: a stolen instruction inherits a short-lived, tightly scoped credential rather than a master key.
The same role-based thinking applies to what the agent can see and do. On the Convex Insurance work the access model was enforced with RBAC, so a component could only touch the data its role permitted. Put an agent inside that boundary and injection cannot make it exceed the role. This is where security and the AI build meet, and why we treat security and compliance as part of designing AI agents and LLM products, not a review at the end.
Human-in-the-loop for high-impact actions
Some actions are not worth automating end to end no matter how good the guardrails are: sending money, deleting records, emailing customers, changing permissions, deploying to production. For these, put a person in the path. The agent proposes the action and shows exactly what it is about to do, and a human approves it before it executes.
The point is to draw the line by impact rather than by confidence. A model that is confident is not a model that is correct, and an injected instruction will be delivered with as much confidence as a legitimate one. So the trigger for review should be the consequence of the action, measured by what it touches and whether it can be undone, not the model's certainty. Reversible, low-stakes actions can run automatically. Irreversible or sensitive ones wait for a click. Make the approval prompt show the resolved values, not the model's summary of them, so the reviewer sees the real recipient and the real amount.
Input and output validation, and guardrail models
Filtering still has a place, as long as you are honest about what it buys you. You can screen input for known injection patterns and screen output before it leaves your system, for example to catch a response that contains a system prompt or a credential. A separate guardrail model, a smaller classifier that scores a request or a response, adds another opinion to the chain.
Be clear about the limit. These are probabilistic detectors against an adversary who can rephrase indefinitely, so they catch the careless and the known, and they will miss novel or obfuscated attacks. Treat them as one layer that raises the cost of an attack, never as the control that makes the model safe to wire into a privileged tool. If a guardrail is the only thing standing between an injected instruction and a destructive action, you have a single point of failure wearing a safety label. Their value is real but bounded: they reduce noise and buy detection, while the tool restrictions and least-privilege credentials do the load-bearing work.
Gateway-level PII redaction
There is a related exposure worth handling at the same boundary. Whatever the model sees can end up in a provider's logs, in a later completion, or pulled back out through injection. So control what leaves your perimeter before it reaches an external model.
Route provider calls through a gateway and redact or tokenise sensitive data on the way out: names, account numbers, anything covered by your data-protection obligations. The model works on placeholders, and you map them back to real values inside your own boundary when you act on the result. This came up directly on the British Council AiBC build, where GDPR and data residency shaped how requests were handled, and the practical version of "data minimisation" is to make sure the model is given the least it needs to do the task. It limits what an injection can exfiltrate, and it keeps the compliance story honest.
Monitoring and evaluation
Prevention is never complete, so you also need to see what is happening and keep testing the defences. Log the tool calls an agent makes, the parameters it passed, and the content it retrieved, so that when something looks wrong you can reconstruct the path. Alert on the signals that matter: a spike in refused tool calls, a sudden change in which tools an agent reaches for, output that trips an output filter.
Then test on purpose. Keep a growing set of injection cases, direct and indirect, and run them against the system in your evaluation pipeline so a regression in a guardrail or a tool boundary shows up before a release rather than after an incident. Add every real attempt you observe to that set. The defences only stay honest if you keep attacking them yourself.
Untrusted output
Model output is treated as untrusted input. It is never interpolated into a query, command, request, or another agent's prompt without validation.
Tool allowlists
A closed set of typed tools, no arbitrary fetch, shell, or code execution. Every parameter is schema-validated and value-constrained before the call runs.
Least privilege
The agent has its own short-lived, tightly scoped identity. A hijacked session inherits one user's narrow permissions, not a master key.
Human approval
High-impact and irreversible actions pause for a person, who sees the resolved values rather than the model's summary of them.
Guardrails and validation
Input and output filters and a classifier add detection, treated as one layer that raises attacker cost, never as the control that makes a tool safe.
Gateway redaction
Sensitive data is redacted or tokenised before it reaches an external provider, so the model works on placeholders and an injection has less to exfiltrate.
Why one filter is not enough
- The attack surface is the whole context window, and the legitimate instructions and the malicious ones are the same kind of text.
- Indirect injection arrives inside retrieved documents and tool responses, so it bypasses any control that only inspects user input.
- A classifier faces an adversary who can rephrase without limit, so it catches the known and misses the novel.
- Independent layers mean a single bypass does not reach a privileged action, because the tool boundary and the credential scope still stand.
A defence-in-depth checklist
Pulling it together, here is the set of controls we put in place before an agent with real permissions goes near production. No single item on its own is the answer. The protection comes from running them together, each assuming the others can fail.
Controls to have in place
- Treat every model output as untrusted: validate and escape before it reaches a query, command, request, or downstream agent.
- Expose a closed allowlist of tools with strict, typed, value-constrained parameters, and no general-purpose fetch, shell, or code-execution tool.
- Give the agent a least-privilege identity with short-lived, tightly scoped credentials and role-based access, so a hijack has a small blast radius.
- Require human approval for high-impact and irreversible actions, gated by consequence rather than by the model's confidence, showing resolved values.
- Run input and output validation and a guardrail model as detection layers, knowing they raise attacker cost rather than guarantee safety.
- Redact or tokenise sensitive data at a gateway before it reaches an external provider, following data minimisation and residency rules.
- Log tool calls, parameters, and retrieved content, alert on anomalies, and run a growing set of injection cases in your evaluation pipeline.