Agent Reliability

inspectharnesstoolingevaluationopen-source

Inspect (UK AI Security Institute evaluation framework)

The open-source evaluation framework from the UK AI Security Institute: composable solvers and scorers, sandboxed execution of untrusted model code, agent primitives, and a registry of 200+ prebuilt evals. The closest thing agent evaluation has to shared infrastructure — METR's Hawk platform builds on it — and the default answer to "should we write our own harness runner?" (usually no).

Why this wins its question: Answers the build-vs-adopt question with named capabilities and a named production adopter, where most coverage of Inspect is a feature list without the decision context.

Claims

Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.

  1. Inspect is an open-source LLM evaluation framework by the UK AI Security Institute (GitHub UKGovernmentBEIS/inspect_ai) providing solvers, scorers with both text-based and model-graded options, and over 200 prebuilt evaluations.

    confidence 0.9Inspect — evaluation framework for large language models · primary

  2. Inspect runs untrusted model code in sandboxes via Docker, Kubernetes, Modal, Proxmox and other backends through an extension API, and ships agent primitives — ReAct, multi-agent composition, and driving external agents such as Claude Code, Codex CLI and Gemini CLI.

    confidence 0.9Inspect — evaluation framework for large language models · primary

  3. METR's Hawk large-scale agent-evaluation platform is built on Inspect infrastructure — third-party adoption at frontier-evaluation scale.

    confidence 0.85METR — Model Evaluation & Threat Research · secondary