How to evaluate an AI agent before launch: 75% per run means 42% three runs in a row

An agent that succeeds 75% of the time on a single run passes the same task three times in a row only about 42% of the time. Anthropic uses this math for customer-facing agents, where users expect the same result every time. It also explains why a good demo says little about an agent in production: the demo shows one success, while a business runs the same task hundreds of times.

AI agent evaluation before launch needs four things: 20–50 real tasks, several runs of each, checks on the state the agent leaves behind, and an LLM judge that has been calibrated against a human. Below is how each part of an LLM eval works for agents, where it breaks, and a checklist to run before you sign off on a launch.

01 · A demo measures the lucky run

A vendor demo, a pilot screenshot and most leaderboard numbers share one metric: did the agent solve the task once. In eval terms that is pass@1, or pass@k when the best of several attempts is shown.

Production asks a different question. A support agent, a document check or a CRM update repeats every day, and each failure is an incident. The metric for that is pass^k: the probability that all k attempts succeed. τ-bench introduced it in 2024 for agents that talk to users and change records.

The two metrics drift apart fast. With a per-run success rate p, pass^k = p^k:

• 75% per run → 42% for three runs in a row → under 6% for ten;
• pass@k moves the other way and approaches 100% as k grows.

Fresh data shows the gap on real workflows. Thinkingbox (arXiv, August 2026) runs each of 507 policy-bound tasks 20 times. The strongest model, Claude Opus 5, solves 66.5% of tasks on the first try and 47.5% on all 20 tries. For Qwen3.8-27B the gap is wider: 51.7% on the first try, 89.4% on at least one of 20 tries and 10.9% on all 20. The authors put it plainly: retries “can expose successful trajectories without yielding dependable execution”.

Rule of thumb: if a task repeats in production, report pass^k next to pass@1. If a vendor shows only best-of-N, ask for the per-run rate.

02 · What an agent eval is made of

To build pass^k into a process, you need a few parts with clear names. Anthropic’s engineering team uses this vocabulary:

• Task: input, environment and a success criterion.
• Trial: one run of a task. Agents are not deterministic, so each task gets several trials.
• Grader: the check. Code-based (tests, exact match, final state), model-based (rubric, pairwise comparison) or human.
• Transcript: every step, tool call and intermediate result. It tells you whether the agent failed or the grader rejected a correct answer.
• Harness: the code that runs tasks, stores transcripts and computes metrics.

A beginner needs only the first three. Transcripts and the harness become the main work once the set grows past a few dozen tasks.

03 · Where the tasks come from

With the parts named, the next question is what goes into the set. Anthropic’s advice: “20-50 simple tasks drawn from real failures is a great start.” Real failures matter because invented test cases tend to repeat the developer’s assumptions.

Two effects show up as soon as people start grading:

1. Criteria drift. The EvalGen study (UIST 2024) found that “grading outputs helps users define criteria”. Write a first rubric, grade 20–50 real outputs, then rewrite the rubric from the notes.
2. Contamination. If you tune the agent on the same tasks you test on, the score stops measuring generalisation. The Evaluation Context Protocol draft (August 2026) calls for “strict separation of developmental iteration sets from withheld production test sets”.

A reference set also turns quality work into numbers you can manage. On StroyIntellekt, a multi-agent product I own that reviews construction cost documents, we built a reference base of 103,000 federal estimate items and 150+ building codes. Measured on our internal reference set, parsing accuracy rose from 21% to 86%. Without that set the change would have stayed an impression (“parsing got better”). With it, the team had a figure to plan against. Each of the ten specialist agents in that system also has its own acceptance checklist, so a regression can be traced to one agent.

04 · Check what the agent changed

A good task list still fails if the grader reads only the agent’s answer. Thinkingbox shows why: 80.9% of failed trials end cleanly and call a tool that changes data, and 67.2% also end with no explicit error in the final tool response. Judging by the reply and the last tool call, such a failure is invisible.

τ-bench solved this in 2024 by comparing the final database state with an annotated goal. The same idea works in a product:

• the CRM record, file or ticket after the run is compared with the expected one;
• forbidden actions (sending email, deleting rows, payments) are checked as a separate pass/fail;
• the output schema is validated by code, and an invalid output fails the task.

On StroyIntellekt we followed a stricter version of this: code computes, the model explains. Money and arithmetic run in deterministic code, every number in the report links back to its file, sheet and row, and model output that breaks the schema is rejected (fails closed). Such a split makes most grading cheap: sums and links can be checked by code, and only the explanations need a model or a person.

The ECP draft reaches the same conclusion: “code-based graders remain the ideal default for verifying deterministic outcomes”.

05 · An LLM judge (LLM-as-a-judge) needs its own eval

Some outputs cannot be checked by code: the quality of an explanation, the tone of a reply, whether a finding is justified. Here teams use an LLM as a judge. LangChain’s survey of 1,340 respondents (late 2025, a vendor survey of its own audience) found LLM-as-judge at 53.3% of organisations and human review at 59.8%.

A judge brings known biases. The MT-Bench paper (2023) documented “position, verbosity, and self-enhancement biases”. On harder tasks the picture gets worse: in the REFLECT study (May 2026) none of 14 judges, 8 of them proprietary, reached 55% overall accuracy when checking evidence in research-agent trajectories.

So the judge is calibrated like any other component:

1. One principal domain expert labels 30–100+ outputs: pass/fail plus a written critique.
2. The critiques become few-shot examples in the judge prompt.
3. The judge labels the same set. You compute agreement, Cohen’s kappa and, separately, the false-pass rate, the most expensive error.
4. Iterate the prompt until it clears the bar. The ECP draft suggests at least 100 human-labelled examples and kappa of 0.6 or higher.
5. Confirm on a held-out set the prompt has never seen.
6. For pairwise comparisons, run both orders and use a judge from a different model family.

Hamel Husain calls this critique shadowing. At Honeycomb it reached “90% agreement between the LLM and [domain expert] in three iterations”. The number is specific to that product; the method transfers.

Use binary criteria. ECP: judges “must be restricted to binary pass/fail grading based on specific, unambiguous rubrics, rather than relying on abstract 1-to-5 point scales”.

06 · Rewording a prompt moves results more than rerunning it

Once the judge is calibrated, the next source of error is the test itself. The Noise Floor Audit (August 2026) measured both on a function-calling benchmark (BFCL) at temperature 0: rerunning the same prompt barely changes results (flip rates of 0.7–2.7%), while meaning-preserving rewordings of the prompt produce spreads 11 to 58 times larger.

Two practical consequences:

• version prompts like code and compare agent versions on the same frozen set of phrasings;
• include two or three phrasings per task, or a small gain between versions may be noise.

The same study found that malformed output caused 30%, 7% and under 1% of failures on three endpoints from two providers. The average score hid a different failure mode on each.

07 · Turn evals into a release gate

Measurements pay off when they decide something. In my view three gates cover most agent products:

• Per-agent acceptance checklists. In a multi-agent system each specialist agent gets its own checklist, so a regression points to one agent.
• Golden scenarios for model changes. In AgenticCRM, the agentic CRM I’m building with a team of six engineers, 30 eval scenarios certify models: any replacement model has to pass the same 30 scenarios first.
• A weighted profile with floors. Score output on several weighted criteria and set a floor for the critical ones, for example an overall 82 or higher and no critical criterion below 70. An average alone would hide one broken criterion.

Most teams are not there yet. In the LangChain survey 52.4% of organisations run offline evals on test sets and 37.3% run online evals on live traffic. Among teams with agents in production, 94% have observability but 44.8% evaluate that traffic. Traces without grading are logs: they record what happened and leave open whether it was right.

08 · What evals cost and where they stop working

Evals are a budget line. The Holistic Agent Leaderboard spent about $40,000 on 21,730 agent runs, roughly $1.80 a run. pass^k multiplies that by k, and a model judge adds its own tokens. Plan the eval budget next to inference cost from the start.

Three limits to keep in mind:

1. Public benchmarks don’t transfer. In February 2026 OpenAI stopped reporting SWE-bench Verified: 59.4% of the 138 hard tasks it audited had flawed tests, and models showed signs of knowing answers. Benchmarks help you shortlist models; decisions need your own tasks.
2. A set ages. Users change, the product changes, criteria drift. Add every production incident as a new task and review the gold set on a schedule.
3. One expert means one taste. Critique shadowing aligns the judge with one person. For a product with very different user groups, calibrate a judge per group or keep human review on the boundary cases.

09 · Checklist: the eval set before you launch an agent

Start:

• 20–50 tasks from real requests and past failures, with an expected end state for each.
• A held-out set you never use while tuning prompts or tools.
• 3+ runs per task; pass@1 and pass^k reported side by side.

Check:

• Code graders for state, schema, sums and forbidden actions; model or human graders only for what code can’t judge.
• The LLM judge calibrated on 100+ expert labels, kappa ≥ 0.6, false-pass rate tracked.
• Two or three phrasings per task to separate signal from prompt noise.
• Transcripts sampled and read every week.

Stop and fix before launch if:

• pass^3 on high-stakes tasks is below your threshold, even when pass@1 looks good;
• any critical criterion falls below its floor;
• a model or prompt change hasn’t passed the golden scenarios.

If a contractor builds the agent, make the eval set part of the deliverable; without it, don’t sign off on acceptance. The eval set becomes the product’s most valuable test asset: models change every few months, and the set tells you in an afternoon whether the new one is safe to ship. The companion piece on prompt injection cases covers the security side of the same launch decision, and AI agents in product covers how teams move from answers to actions.

Which number does your team look at before an agent ships: pass@1, pass^k or something else?