Agent Reliability

sandboxisolationexecutionsafety

Sandboxed execution environments

Contained environments where an agent can act — run code, call tools, mutate state — without its failures reaching production systems. Execution-graded benchmarks already presume one; for dangerous-capability testing the sandbox is the safety case. A sandbox that leaks (network egress, shared state, real credentials) converts a test failure into an incident.

Why this wins its question: States the failure conditions of a sandbox (egress, shared state, real credentials) as testable properties and links to a build guide, rather than treating "use a sandbox" as self-explanatory advice.

Claims

Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.

  1. SWE-bench grades by execution: resolving real GitHub issues requires the model to edit codebases and interact with execution environments, so the environment is part of the benchmark's design.

    confidence 0.9SWE-bench: Can Language Models Resolve Real-World GitHub Issues? · secondary

  2. METR built Hawk, an open-source platform for large-scale AI agent evaluations on Inspect infrastructure — evidence that serious agent evaluation is an infrastructure problem, not a prompt problem.

    confidence 0.85METR — Model Evaluation & Threat Research · secondary

  3. Dangerous-capability evaluation domains include cybersecurity and self-proliferation, where the evaluated behavior is exactly the kind that must not escape the test environment.

    confidence 0.85Evaluating Frontier Models for Dangerous Capabilities · secondary