Hardening Tool-Using LLM Agents Against Prompt Injection

Once an untrusted span is in the context window, every downstream "decision" by the model is suspect. The only way to keep authorization decisions trustworthy is to make them before the agent reads attacker-controlled text. This is the tool-precommitment pattern.

A trusted planner sees only the user request and developer policy, and emits a structured manifest: allowed tools, parameter scopes, max result counts, allowed destinations, write permissions, and consent thresholds. The runtime treats this manifest as immutable for the session.

Read a session trace and confirm the manifest is fully populated and timestamped before the first untrusted-origin token enters any model's context.

Generic catch-all tools like gmail_api or github_api collapse the read-private / write-external boundary that egress control depends on. A narrowed tool surface makes the validator's job tractable.

Replace broad tools with narrow commands — email_search_readonly, email_draft_reply (no auto-send), email_send_external (step-up auth and DLP). Apply the same split to repos, databases, and chat surfaces.

Enumerate the registered tools and confirm no single tool both reads private data and emits to an external sink.

A tool's blast radius is the union of its credential's scopes. Long-lived broad tokens turn a low-severity injection into account-wide compromise.

Issue per-task ephemeral credentials where the upstream supports it; otherwise pin the narrowest stable scope per tool entry. Bind credentials to the manifest, not the agent process.

For each tool, confirm the granted scopes are no broader than its manifest entry requires, and that grants are reviewed on a fixed cadence.

Runtime tool registration is the cleanest path to tool hijacking: an attacker-controlled span persuades the model to mint a tool that bypasses every other control.

Tool addition requires out-of-band human approval and a code-reviewed manifest change. The runtime rejects model-proposed tool registration, including any "install this MCP server" affordance.

Inject a synthetic "register tool" instruction into a test corpus and confirm the active toolset does not change and that the attempt is logged.

You cannot enforce a taint-flow policy without taint labels. Ingestion is the only place to apply them cheaply and consistently.

Web fetches, email bodies, MCP responses, retrieved chunks, tool outputs, and uploaded files all carry origin, trust, and received_at end-to-end. Labels travel with the span through extraction, summarization, and retrieval.

Inspect any production context payload and confirm every span has a provenance tag traceable to a single ingestion event.

Raw text from an untrusted source is a payload-delivery surface. Typed extracted fields are not — they cannot carry imperative instructions to the planner.

Return {invoice_total, vendor, due_date} rather than the full PDF body; return a parsed thread summary rather than full email HTML. Extractors run in reader roles with no tool authority.

Sample production sessions and measure the fraction of untrusted-origin tokens that reach the planner versus stopping at an extractor; track this ratio over releases.

Zero-width characters, white-on-white text, hidden DOM nodes, and alt-text-only payloads are routine carriers for indirect injection. They are invisible to humans reviewing the source but fully visible to the model.

Normalize ingested HTML, Markdown, and PDF text through a sanitizer that drops or flags hidden content; surface flagged spans to logs rather than to the model.

Feed a known set of hidden-text fixtures through ingestion and confirm they are removed or surfaced with a hidden_content flag.

A single model instance reading attacker-controlled text and choosing tool calls is the textbook confused deputy. Splitting roles makes the privileged decision-maker blind to the payload.

Distinct planner / reader / actor roles. The reader has no tool authority; the actor receives only typed extracted data plus the precommitted manifest, never the raw untrusted span.

Audit the session graph and confirm the actor's context contains no spans labeled trust: untrusted.

Validation inside the model is not validation. The model is the thing being attacked. The check has to live somewhere the attacker cannot reach with text.

A non-LLM policy engine checks: tool allowed, parameters match schema, destination on allowlist, data volume within limits, taint flow allowed, write requires consent. Unknown conditions deny.

Run a red-team harness of out-of-manifest calls and confirm every one is denied and logged with a structured reason.

The EchoLeak chain (CVE-2025-32711) showed that a stack of plausible mitigations still produced zero-click exfiltration because rendered content was treated as output, not as egress. Anything the client fetches on the user's behalf is an egress channel.

The validator and the rendering layer enforce the egress allowlist on Markdown image sources, link auto-fetches, embed URLs, webhook destinations, and DNS-style subdomain patterns — not just on explicit send tools.

Run canary-secret tests (see Detection) that try to exfiltrate via image rendering, link prefetch, and proxy paths; confirm none reach the network.

Consent prose generated by the model is part of the attack surface, not the defense. The user must see ground-truth structured fields, not a paraphrase the attacker influenced.

Generate consent dialogs from runtime structured data — action, destination, data fields, source provenance, whether untrusted content influenced the call — using a non-LLM template. Step-up auth on high-risk classes.

Confirm the consent payload schema is populated by the runtime, not by model output, and that altering provenance flips the dialog.

A corporate proxy is still a network egress. Allowlists scoped only to "external" domains miss the proxy path, which is exactly how the EchoLeak chain reached its sink.

Allowlists are per-task and scoped by the manifest; any internal proxy that fetches arbitrary URLs is treated as an external destination for allowlist purposes.

Attempt an out-of-allowlist fetch in a test session and confirm the validator denies it before the request leaves the process.

Long-term memory is the durable surface for memory poisoning: a single bad write becomes a behavioral default until evicted. Untrusted-origin records must never be read as instructions.

Memory schema includes source, trust, allowed_uses, forbidden_uses. Untrusted-source records default to forbidden_uses: [authorization, tool_selection] and the runtime honors that.

Sample memory records and confirm schema completeness; attempt to use an untrusted-source record to grant tool authority and confirm the runtime refuses.

Unauthenticated write paths into a retrieval corpus are how RAG data exfiltration and policy-override attacks land. Public-web indexes and internal-policy corpora cannot share a trust class.

Every index write is authenticated and attributed; new documents enter a quarantine state and are excluded from high-risk task retrieval until reviewed or aged.

Walk the write path for each index and confirm there is no anonymous or service-account-shared ingestion route.

The model treats convincing retrieved text as authoritative. The runtime must not.

Retrieval returns typed evidence with citations; the validator ignores any tool-permission claim, scope expansion, or policy-override claim that originates in retrieved content. Manifest changes require the planner path, not the retrieval path.

Inject a test chunk requesting new tool authority and confirm no manifest change occurs and the attempt is logged.

Without provenance-linked logs you cannot answer the only question that matters during an incident: which untrusted span caused which action.

Structured logs retain prompts, tool I/O, proposed calls, validator allow/deny reasons, consent payloads, and denied actions, with span-level provenance. Retention long enough for incident response.

Run a tabletop where a tester picks an action from a recorded session and traces it back to a single source span using only the logs.

Canaries catch exfiltration channels you forgot to enumerate, including the rendered-image and proxy paths the EchoLeak chain abused.

Place unique canary tokens in private contexts, memory, and retrieval corpora. Alert on any outbound observation of a canary — including via image fetches, encoded URLs, DNS queries, and proxy paths.

Scheduled canary-egress tests run against a staging tenant and alerts fire on each known channel.

Static-payload "0% ASR" claims are not evidence of robustness. AgentDojo, ASB, and MCPTox baselines show realistic systems sit far from zero, and adaptive attackers move the number further.

Maintain an internal red-team suite that updates per release; report benign utility, utility-under-attack, and targeted attack success rate. Treat regressions in any of the three as release-blocking.

Each release has a published evaluation row with all three metrics and a delta against the prior release.

MCPTox and related work show tool descriptions are themselves an injection surface; an unreviewed description update can quietly redefine what a tool does in the model's eyes.

Pin MCP server versions by hash, sign manifests, review natural-language description diffs in code review, and prohibit imperative instructions in tool descriptions.

Inventory the MCP servers in use and confirm every entry has a pinned hash, a reviewed description diff, and an owner.