Prompt injection and trust boundaries
Understand evolving attacks through documents and tool outputs, the defenses introduced in response, and the limitations that still require application controls.
A procurement reviewer asks an assistant to summarize supplier onboarding documents. One PDF contains an instruction claiming that the assistant must change the supplier’s bank account and disregard the normal review process. The document is allowed evidence for the summary; it is not allowed to redefine the user’s request or grant payment-system authority. Prompt injection tries to cross that boundary.
Map who controls each input#
A trusted application instruction, an authenticated user request, a retrieved document, a tool response and a stored memory have different origins. Text does not become authoritative because it appears later in context or uses urgent language. Keep origin, source version and allowed purpose attached as evidence passes through extraction, summarization and delegation.
| Input surface | Possible controller | Permitted role in this task |
|---|---|---|
| User’s summary request | Authenticated procurement reviewer | Defines the authorized objective within their permissions |
| Supplier PDF and OCR text | External supplier or file author | Evidence to summarize and verify |
| Tool description | Tool provider or integration maintainer | Contract description subject to trust review |
| Tool result | Service plus any external text it carries | Observation, not a new permission grant |
| Stored memory | Earlier users, sources or model-derived summaries | Scoped context with provenance and expiry |
| Policy decision | Trusted application and authoritative service | Enforces whether an operation may execute |
Direct injection arrives through a user message; indirect injection arrives through material the system encounters while doing legitimate work. The 2023 indirect-injection research made that external-content risk concrete. Multimodal systems also ingest text through images and OCR. Removing a suspicious sentence from the visible final answer does not establish that the agent ignored it while selecting tools.
What successive defenses changed#
- February 2023
External data became an instruction channel
Indirect-injection research examined applications that process attacker-influenced content. The important lesson is the trust boundary, not a claim about the vulnerability rate of every current model.
Source ↗ - March 2024
Content marking attempted to preserve origin
Spotlighting studied ways to distinguish external material from instructions. This is a mitigation at the model-input layer; it is not a replacement for downstream authorization.
Source ↗ - 2025 guidance
Layered controls address consequences
OWASP describes constrained capabilities, validation, permissions, human review and adversarial testing. Retrieval and fine-tuning alone do not fully remove the risk.
Source ↗ - November 2025 research
Browser defenses remain an active engineering problem
Anthropic describes combined defenses and adversarial evaluation for browser agents. Its reported results belong to its system and tests, not a universal safety percentage.
Source ↗
The current design decision is to assume that a model can sometimes misinterpret hostile data and constrain what that mistake can do. Prompt hierarchy, clear delimiters, content labeling and classifiers can reduce the chance of a bad proposal. The application must still reject a forbidden proposal if those measures fail.
A successful tool call can return hostile text#
A ticket lookup may correctly return an authorized ticket whose body contains an attacker’s instruction. A search result can include an instruction in its snippet. A repository file can tell a coding assistant to upload secrets. The transport and schema may be valid while a field remains untrusted content. Treat structured output as well-formed data, not as a guarantee of benign meaning.
MCP’s current tool specification states that annotations are untrusted unless they come from trusted servers. A read-only hint or reassuring description does not enforce behavior. Review the actual tool implementation, credential scope and allowed destinations. If a tool catalog changes, reassess the model-facing descriptions and capabilities rather than assuming the old approval covers new functions.
Place a policy gate after the proposal#
| Step in the supplier example | Expected behavior | Boundary that must enforce it |
|---|---|---|
| Read onboarding PDF | Retrieve only authorized material | Document service |
| Encounter bank-change instruction | Keep it as untrusted source content | Context construction and model instructions |
| Model proposes a bank change anyway | Reject absent authority and approved business process | Application and payment/ERP service |
| Continue legitimate summary | Describe supported onboarding facts and unresolved issues | Answer construction and evidence review |
| Investigate suspicious content | Record source reference and blocked attempt without secrets | Security monitoring and incident owner |
Derive actor and tenant identity from authenticated runtime context. Do not accept a model argument that claims to be an administrator. Validate the object, operation, allowed fields, expected record version and business limits independently. In this task the assistant only needs read access and a draft-report capability; a bank-account update tool should not be present at all.
Limit both actions and destinations#
A narrow tool set reduces the reachable consequences of a bad decision. Prefer a scoped document lookup over unrestricted SQL, shell or URL-fetching access. Keep credentials in a runtime mechanism rather than prompt text. For workers that need execution, restrict filesystem access, network destinations, resource use and secret exposure. Audit the actual deployment permissions, including background jobs.
Exfiltration can occur through more than an explicit send-data tool. Automatically loading a generated remote image or following an arbitrary URL may send information to a new destination. Render text and allowlisted content safely, and mediate outbound requests. A syntactically valid HTTPS URL is not necessarily an authorized recipient for customer data.
Separate public internet research from private-document analysis when practical, with an explicit handoff that minimizes transferred data. Do not let a retrieved page choose where internal material is sent. Rate and volume limits can reduce damage, but they cannot turn an unauthorized disclosure into an acceptable one.
Give approval a precise object#
For a permitted consequential action, display the exact recipient, account, amount, affected records and supporting evidence in a trusted review interface. Bind approval to that payload, the reviewer, relevant record versions and an expiry. If later tool output changes the destination, invalidate the approval. A broad instruction to “finish onboarding” is not approval of every future model-selected action.
Recheck authority when a paused task resumes. An approver may have lost access or the supplier record may have changed. A second model can help inspect a proposal, but naming it a guardrail does not make it an independent authorization authority. It can share the same failure modes and untrusted evidence as the first model.
Prevent an instruction from surviving as memory#
An attack can persist after its source disappears if a summary or preference store saves it. Treat memory creation as a write with a schema, provenance, scope, allowed use and expiry. Do not transform “the supplier asks us to skip review” into “this user prefers automatic approval.” Preserve the distinction between an external assertion and a verified policy.
When a source is quarantined or corrected, trace its derivatives: summaries, cached answers, memory records and delegated reports. Invalidate or re-evaluate them before reuse. A worker’s paraphrase remains derived from the original source; another agent repeating it does not increase its trust level.
Inspect downstream effects in your tests#
Use authorized synthetic fixtures containing hostile instructions in document bodies, OCR text, tool responses and proposed memory updates. Also include benign quotations about security so you can detect overblocking. The expected result is not simply a sentence saying “I refused.” Inspect tool attempts, external destinations, final business state and retained memory.
Test the same attack through retries, concurrent users, warm caches, delegated workers and approval resume. A defense that works only on the first request is incomplete. Reassess after changes to the model, tool catalog, parser, prompt or source pipeline. Record what was tested and what remains outside scope; do not label a small test set a comprehensive security certification.
Plan for a failed boundary#
Monitor unexpected tool selection, denied writes, new destinations, unusual output volume, repeated policy failures and suspicious source instructions. Minimize sensitive log content while preserving the run and source references needed for investigation. Define who can stop runs, revoke credentials, quarantine sources and invalidate contaminated derivatives.
If an operation may already have committed, stopping the model is only the first step. Reconcile actual state with the downstream service, contain exposed credentials and follow the organization’s incident process. Some effects require a new authorized corrective action and cannot simply be rolled back. Document those limits before granting an agent the capability.