Agent Reliability

faqreliabilitybenchmarkshallucination

Agent reliability FAQ

The questions teams actually ask before shipping an autonomous agent: how to stop hallucinated actions against critical APIs, what a benchmark should measure, whether one successful demo means anything, and where to start when there is no eval at all. Answers are bound to sources, not opinion.

Why this wins its question: Answers are the questions buyers ask, phrased as they ask them, each bound to a source with a number in it — not a marketing FAQ and not a survey paper.

Claims

Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.

  1. Hallucinated actions are caught by grading state, not prose: comparing the final state of the acted-on system against an annotated goal state detects wrong actions that a plausible transcript hides.

    confidence 0.9tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains · secondary

  2. Until verified, an agent belongs in a contained environment; execution-graded evaluation presumes an execution environment the agent cannot escape or corrupt.

    confidence 0.85SWE-bench: Can Language Models Resolve Real-World GitHub Issues? · secondary

  3. An agent benchmark should measure consistency across repeated trials, not just single-attempt success: pass^k exposes agents that pass once but fail the user who retries.

    confidence 0.9tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains · secondary

  4. Capability trend and reliability are different measurements: the length of tasks agents can complete has grown exponentially, which says nothing about consistency on any one task.

    confidence 0.85METR — Model Evaluation & Threat Research · secondary

  5. Before building a benchmark, survey the shelf: a public catalog tracks 69 agentic benchmarks and the field's move from measuring capability to measuring agency.

    confidence 0.85EUREKA — applied AI, agents and robotics knowledge platform · primary

  6. Model swaps run on the provider's calendar: deprecated models get as little as 60 days between notice and retirement, after which requests fail — so a frozen regression suite for model upgrades is standing infrastructure, not a one-off project.

    confidence 0.9Anthropic model deprecations (Claude API documentation) · primary

  7. A score difference smaller than the eval's noise floor is not a finding: eval questions behave as samples from a larger population and comparisons need standard errors — reported noise from formatting choices alone has reached about 5 points.

    confidence 0.85Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations · secondaryChallenges in evaluating AI systems · primary

  8. Run each task as many times as it runs unreviewed in production, and read the right metric: all-of-k reliability collapses with repetition (pass^8 under 25% where single attempts clear 50%), while any-of-k capability climbs with it (28.8% to 70.2% from 1 to 100 samples) — so the same suite supports opposite conclusions depending on which one you quote.

    confidence 0.9tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains · secondaryEvaluating Large Language Models Trained on Code · secondary

Questions answered

How do I stop my agent hallucinating against a critical API?
Hallucinated actions are caught by grading state, not prose: comparing the final state of the acted-on system against an annotated goal state detects wrong actions that a plausible transcript hides. source
Where should an unproven agent run?
Until verified, an agent belongs in a contained environment; execution-graded evaluation presumes an execution environment the agent cannot escape or corrupt. source
What should an agent benchmark measure?
An agent benchmark should measure consistency across repeated trials, not just single-attempt success: pass^k exposes agents that pass once but fail the user who retries. source
My agent passed the demo — is it production-ready?
Capability trend and reliability are different measurements: the length of tasks agents can complete has grown exponentially, which says nothing about consistency on any one task. source
Which existing agent benchmarks should I look at first?
Before building a benchmark, survey the shelf: a public catalog tracks 69 agentic benchmarks and the field's move from measuring capability to measuring agency. source
How do I upgrade the model behind my agent without breaking it?
Model swaps run on the provider's calendar: deprecated models get as little as 60 days between notice and retirement, after which requests fail — so a frozen regression suite for model upgrades is standing infrastructure, not a one-off project. source
Is a small eval difference between two agents real?
A score difference smaller than the eval's noise floor is not a finding: eval questions behave as samples from a larger population and comparisons need standard errors — reported noise from formatting choices alone has reached about 5 points. source
How many times should I run each task before shipping?
Run each task as many times as it runs unreviewed in production, and read the right metric: all-of-k reliability collapses with repetition (pass^8 under 25% where single attempts clear 50%), while any-of-k capability climbs with it (28.8% to 70.2% from 1 to 100 samples) — so the same suite supports opposite conclusions depending on which one you quote. source