Designing an agent sandbox that actually contains failures
Requirements and checks for an execution environment where an agent can fail safely: no default egress, ephemeral seeded state, fake credentials that look real, a bounded blast radius, and an observation channel the agent cannot write to. Includes a containment checklist you can test the same way you test the agent.
Why this wins its question: Turns "run it in a sandbox" into a falsifiable checklist — each containment property is stated as a test you can run against the sandbox itself, in the same spirit as guardian falsification.
Key takeaways
- A sandbox with egress, shared state or real credentials is production with extra steps.
- Every containment property must be testable — and falsified once before it is trusted.
- Keep the grader and telemetry outside the boundary, read-only; an agent that can edit its transcript passes any transcript-graded eval.
- Adopt existing evaluation infrastructure; build only the fixtures and graders that encode your domain.
Claims
Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.
Chaos engineering requires that the fallout from experiments be minimized and contained; blast radius is a design input, not an afterthought.
Execution-graded benchmarks presume a working environment: SWE-bench tasks require editing real codebases and interacting with execution environments to verify resolution.
Large-scale agent evaluation is infrastructure work — METR's Hawk platform, built on Inspect, exists specifically to run agent evaluations at scale.
Dangerous-capability evaluations cover cybersecurity and self-proliferation — behavior whose escape from the test environment is itself the incident being prevented.
What the sandbox is for
An agent under test will do the wrong thing — that is the point of testing it. The sandbox's job is to make the wrong thing observable and cheap. If a test failure can reach a production API, mutate shared state, or exfiltrate a real credential, you do not have a sandbox; you have production with extra steps.
Containment properties (each one testable)
1. No default egress. The environment reaches only an explicit allowlist of endpoints, all of them fakes or fixtures. Test it: run an agent whose task is to call an unlisted endpoint; the call must fail visibly. 2. Ephemeral, seeded state. Every run starts from a fixture and throws its world away afterwards. Test it: run twice, diff the starting states; any difference is a leak. 3. Fake credentials that look real. The agent must not learn it is being tested from the shape of its secrets, and a leaked test secret must be worthless. Test it: plant the credential in output and verify the alarm fires and nothing real was exposed. 4. Bounded blast radius (claim c1). Enumerate, in writing, the worst thing a fully adversarial agent could do inside the environment. If the answer includes any production noun, redesign. 5. An observation channel the agent cannot write to. Telemetry and grading read from outside the environment. An agent that can edit its own transcript can pass any transcript-graded eval.
Grading inside the sandbox
Execution-based grading is the reason the environment exists (claim c2): the sandbox hosts the codebase, the database, or the API fixture whose final state the harness compares against a goal state. Keep the grader outside the sandbox boundary and read-only.
Build vs adopt
Platform work at evaluation scale is real engineering (claim c3). Adopt existing open infrastructure (Inspect-based platforms such as METR's Hawk) when your tasks fit; build only the fixtures and graders that encode your domain. The capability ceiling of your sandbox should match the capability you are probing — evaluations that probe cybersecurity or self-proliferation behavior (claim c4) treat containment as the primary deliverable, and so should any team whose agent holds credentials to critical APIs.
Checklist
- [ ] Unlisted egress fails, visibly, with an alarm.
- [ ] Two consecutive runs start from identical state.
- [ ] Test credentials are worthless outside the sandbox.
- [ ] Written worst-case contains no production noun.
- [ ] Grader and telemetry are read-only from outside.
- [ ] The sandbox itself has been falsified: you have watched each
property fail once.