Prompt Injection: How It Works and How to Defend Your App

Prompt injection happens when an AI application treats attacker-controlled content as instructions it should follow. The content might be a direct message, a retrieved document, a web page or a tool response. The attacker tries to redirect the system away from its intended task, sometimes towards exposing data or taking an unauthorised action.

The important distinction is architectural: models process instructions and data through overlapping language channels. A sentence inside a document can look like an instruction even though the application only intended the model to summarise it. Stronger prompts help communicate intent, but reliable defence requires permissions and data boundaries outside the model.

Map the attack surface before rewriting prompts

Distinguish direct injection, supplied through an interaction with the assistant, from indirect injection, embedded in material the assistant retrieves or reads. A support agent summarising an uploaded attachment can encounter hostile instructions without the attacker ever opening the chat interface.

List every source entering model context: user text, document titles, search snippets, repository files, image text and tool results. Record who controls each source and what actions it could influence. A trusted retrieval service can return untrusted content; transport authenticity does not make the document's instructions authoritative.

Identify valuable targets. These might include confidential context, external network requests, account changes or persistent memory. An assistant that only rewrites public copy has a different risk profile from one that can send messages or query customer records.

Read the OWASP prompt injection guidance when building this inventory. Treat instruction hierarchy as one useful defence, not proof that hostile lower-trust content cannot affect model behaviour. Document the actions your application must prevent even if the model follows the attack completely.

Keep authority in application code

The server should decide authorisation, not the model. Authenticate requests, determine the user's workspace and constrain database queries before returning records to the assistant. A model-generated customer identifier must not expand the set of records that user can access.

Apply least privilege to every tool. Separate read-only retrieval from write actions and give each operation only the credentials it needs. Prefer a narrowly defined action such as updating an approved draft over a general shell or arbitrary HTTP tool. Limit arguments, execution time and result size.

Check permissions again at execution time. A generated action plan can become stale, and a permission checked earlier in a conversation may no longer apply. The model can propose an operation, but the application must independently decide whether it is allowed now.

Never place secrets in model context merely because a system prompt says not to reveal them. Keep credentials in server-side integrations and return only the data required for the task. The OWASP authorisation guidance remains relevant: AI integration does not replace ordinary access control.

Separate untrusted evidence from executable actions

Label retrieved content as untrusted evidence and keep its source metadata attached. Clear delimiters and explicit instructions can reduce confusion, but they are not a security boundary. An attacker can imitate delimiters or invent a convincing claim that the source has higher authority.

Use constrained interfaces between analysis and execution. For example, let a summarisation step return candidate facts and source identifiers, then have ordinary application code validate those identifiers. Do not turn arbitrary generated strings into database queries, executable code or outbound destinations.

A useful security review prompt for the implementation team is:

Assume the retrieved document fully controls the model's next answer.
Can that answer read another tenant's data?
Can it invoke a write tool without current authorisation?
Can it send context to an arbitrary external destination?
List the server-side check that blocks each action.

Protect egress boundaries as well as database access. Restrict outbound destinations, validate redirects where applicable and prevent arbitrary fetch tools from accessing internal services. Validate URLs using a proper parser and a maintained policy rather than matching a trusted domain as a substring.

Rendering is another boundary. Treat generated Markdown and links as untrusted output, disable raw HTML where possible and sanitise with a suitable library if HTML is necessary. Follow the MDN content security policy guide for browser-side defence in depth, while recognising that CSP does not stop server-side tool misuse.

Require meaningful approval for consequential actions

A confirmation dialog is useful only if it shows the actual operation. Present the destination, affected records and significant parameters in trusted application UI. Do not let a generated explanation conceal different underlying tool arguments.

Bind human approval to a specific action payload and invalidate it if those arguments change. For a message-sending tool, show recipients and final content. For deletion, show the exact resources and scope. A vague approval to continue the conversation is not authorisation for every later action.

Build safe previews and dry runs where possible. Let users review a proposed change without committing it, and favour reversible operations over permanent ones. Log the approved action and outcome without unnecessarily storing sensitive source material.

Keep approval requirements proportional to consequence. Requiring a modal for every harmless lookup causes fatigue and trains users to click through. Reserve interruption for meaningful risk, while applying silent permission checks to every operation. The AI SDK workflow guide explains how tool boundaries fit within a server-owned generation pipeline.

Test hostile workflows and contain failures

Build an adversarial test set covering direct messages and indirect sources. Include documents that ask the assistant to ignore its task, fabricate authority, reveal hidden context or send data elsewhere. Judge the resulting application behaviour, not whether the assistant politely describes the attack.

Use the API latency tester against a safe test endpoint to establish response timing before and after adding validation or policy checks. Use synthetic requests and keep production credentials out of shared tests. Timing measurements assess operational overhead; they do not establish that an injection defence works.

Add these cases to your AI regression checks, including assertions that forbidden tools were not called and cross-tenant records were not returned. Test multi-step conversations, because malicious instructions can influence a later action rather than the immediate answer.

Plan incident containment before release. Provide switches to disable write tools, revoke integration credentials and isolate affected workflows. Preserve request identifiers and security-relevant events with access-controlled retention. If an attack succeeds, repair the application boundary that permitted the consequence rather than only appending another warning to the prompt.

Key takeaways

• Treat documents, tool results and user messages as potential injection sources.
• Enforce authorisation and least privilege outside the model.
• Restrict tool arguments, outbound destinations and rendered output.
• Bind approvals to explicit, reviewable action payloads.
• Test prohibited consequences and prepare containment controls before launch.