Prompt injection

Prompt injection happens when content the agent reads contains instructions that the agent follows against the wishes of its owner. A web page, an email, a support ticket or a document can contain text such as an instruction to ignore earlier rules or to send data somewhere. A model cannot always tell that text from its real instructions, so the risk has to be handled in the harness.

The risk grows with the agent's reach. An agent that only summarises has a small exposure. An agent that can send messages, change records or call external services has a large one.

In practice

I treat every piece of retrieved or external content as data, never as instructions. The harness keeps the owner's instructions separate from the content, labels the content as untrusted and restricts which tools can run while untrusted content is in context. Any tool with side effects needs a check that does not depend on the model's reading of the content.

I test with injected examples in the evaluation set. The question is not whether the model resists every attack, which it will not, but whether the harness stops the damage when it does not.

Example

[Composite] A research agent read supplier web pages. One page contained a hidden instruction to email the agent's notes to an outside address. The email tool was blocked for the rest of any task that had read external content, so the instruction had no effect. The incident went into the evaluation set as a permanent case.

Common mistakes

• Assuming the model will ignore instructions inside documents.
• Giving an agent write access and external reach in the same task.
• Treating the defence as a prompt sentence. A sentence in the prompt is not a control.