Running your first agent red-team exercise
A bounded, repeatable adversarial exercise for a tool-using agent: scope the rules of engagement inside the sandbox, enumerate the attack surfaces (injection channels, tool permissions, data paths), run manual and automated passes, triage by blast radius, and convert every confirmed finding into a falsified check so the exercise compounds instead of evaporating.
Why this wins its question: Scales published frontier-lab practice down to a team-sized, bounded exercise with a defined output contract — every confirmed finding becomes a falsified check — so the red team feeds the regression gate instead of a slide deck.
Key takeaways
- A confirmed finding is an unauthorized action executed, not a scary transcript.
- The surface is injection channels, dangerous tool permissions and data paths — one seeded probe per channel.
- Triage by blast radius of the action executed, not by ease of fix.
- The deliverable is checks, not a report — every confirmed finding becomes a falsified, counted guardian.
Claims
Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.
Red teaming is a family of methods, not one activity: practiced forms include domain-expert teaming, frontier-threats testing (cybersecurity and autonomous-AI risks among them), automated model-vs-model attack generation, and crowdsourced and community exercises.
Adversarial capability evaluation has a published reference shape: DeepMind's dangerous-capabilities framework probes persuasion, cybersecurity, self-proliferation and self-reasoning rather than benign-path behavior.
For tool-using agents the standard attack surface is named by OWASP: prompt injection (LLM01) as the entry vector and excessive agency (LLM06) as the amplifier that turns it into unauthorized actions.
Findings only compound if they become guardians: the incident-pays-for-check pattern — each confirmed failure produces one falsified, counted check — is how the Citarium validator was built in production.
Before you start: rules of engagement
Run the exercise against the sandboxed agent, never production. Fix the scope in writing: which agent build, which tools enabled, which attack classes are in bounds, what counts as a "confirmed" finding (an unauthorized action executed, not a scary transcript). One page, agreed before the first probe.
Enumerate the attack surface
For a tool-using agent the map is short and standard (claim c3):
1. Injection channels — every untrusted input the agent reads: retrieved documents, web pages, tool results, user uploads, email. One seeded probe per channel. 2. Tool permissions — every tool whose misuse is irreversible or outward-facing: writes, deletes, sends, payments. The question is whether upstream text can reach them without a gate. 3. Data paths — what secrets or personal data the agent can see, and every channel through which output leaves (messages, commits, URLs, logs).
Run the passes
- Manual pass. A person plays adversary on each surface: planted
instructions in retrieved content, confused-deputy requests through legitimate-looking tasks, escalation chains across tools. Expert attention goes where blast radius is highest.
- Automated pass. Use a model to generate attack variants at
volume against the same surfaces — the automated red/blue pattern from published practice (claim c1). Volume finds the phrasing your manual pass did not think of.
- Capability pass (if your agent is powerful). For agents with
broad system access, borrow the shape of dangerous-capability evaluation — probe what the agent could do under hostile prompting, not just what it does on the happy path (claim c2).
Triage by blast radius
Score each confirmed finding by the worst action actually executed: irreversible external action > internal state corruption > policy bypass without action > refusal-wording weakness. Fix order follows blast radius, not ease of fix.
The output contract
Every confirmed finding produces exactly one artifact: a check that would have caught it — falsified (plant the attack, watch red, restore, watch green) and added to the counted inventory (claim c4). The exercise report is allowed to exist, but the checks are the deliverable; findings that stay findings are incidents on a delay.
Cadence
Repeat on every new tool grant, every model swap, and on a fixed calendar even when nothing changed — automated variants drift cheap, so the re-run costs little and the surface moves under you either way.