State of AI 2026 for agent builders: 10 findings, from 100% vs 12.3% to the improvement loop
What the State of AI Report 2026 means for teams that build AI agents: completion vs partial credit, the harness, security, growth and a loop to improve agents.
The same Opus 5 scored 100% on four structured accounting tasks and 12.3% on 100 simulated banking assignments, where the workbook and the delivery both had to be right. That gap runs through the State of AI Report 2026 by Air Street Capital, published on 8 October: free, 244 slides, CC BY.
Read as a builder, the report has one message: the gains now come from the system around the model, and so do the failures. Below are ten findings with their numbers, what each means for a team that ships agents, and where the data stops.
01 · High scores, unfinished work
Benchmarks with partial credit reward an agent for each requirement it meets. Customers pay for finished work.
- FrontierChallenge, 97 scientific workflows: GPT-5.6 Sol scored 87.9 out of 100 on requirements and fully completed 20.6% of tasks.
- EarthVerse, 405 natural-hazard investigations: the same model got 82.1% of required answer items right; 34.8% of tasks reached the 95% threshold.
- OSWorld 2.0, office computer tasks: Opus 5 earned 77.7% partial credit and completed 44.3%.
An expense claim with every field correct is still unfinished if the agent never submits it or misses the approval.
Source: State of AI Report 2026, slide 42, CC BY 4.0.
What to do: count a task as done only when its last step is verified, and report that number next to any average score. The same logic is behind pass^k, the share of tasks an agent passes on every run.
02 · The business metric is a correctly completed job
Capability varies sharply inside one profession. Opus 5 scored 100% over 20 runs on four structured accounting tasks, and passed 12.3% of ATLAS-Finance’s hundred simulated banking assignments, which require both a correct workbook and proper delivery.
In one failed assignment the model passed its own checks, because those checks used the same wrong financial assumption. A self-check built by the agent is not an independent test.
What to do: put an outside check on the outcome (a second source, a rule, a person) for any task with money or customers attached.
03 · High returns, weak screening
Long-running work needs an agent to manage resources, judge risk and learn from earlier decisions. The report’s simulated markets show how far that is from done.
- KellyBench, a simulated season of Premier League betting with £100,000: all 12 tested models lost money on average over five runs, and six went bankrupt at least once.
- E-CommerceBench, 18 models running up to four stores for 365 simulated days: the top earner, GPT-5.6 Sol, finished with CNY 1.43 million in assets and directed 18.48% of its order spending to fraudulent suppliers. Opus 4.7 directed 0.12%.
- Sixteen of the 18 models showed no clear improvement in bargaining on repeat purchases.
These are simulations with their own rules; they do not measure real business profit. They do show that a high final balance can sit next to poor supplier screening. Judge an autonomous agent by its losses, fraud exposure and learning as well as its result.
04 · The harness moves the numbers
The harness is the software around the model: tools, context, state and feedback on failed attempts. In a controlled coding study, GLM-5.1 with fixed weights went from 52.5% with a minimal setup to 56.5% improved and 65.5% with the full setup, on 100 SWE-bench Verified tasks. That is 13 percentage points, about 25% relative.
Different setups also solve different problems. Researchers at Meta, Duke and UC Davis evolved two specialised harnesses and routed each new problem to one of them. With Gemini 3 Flash held fixed, the router reached 62% on a math evaluation against 46% for Meta-Harness; the two branches together could solve 67%.
Source: State of AI Report 2026, slide 8, CC BY 4.0.
What to do: compare models inside the same harness, and compare harnesses with the same model. Otherwise you measure the environment as much as the model. The trade-offs of splitting work between agents are in one agent or many.
05 · Retest the harness after every model upgrade
Some harness features exist to cover a weakness that a newer model no longer has. The report’s agents section describes Claude Code removing 80% of its system prompt for advanced models without a measurable loss on coding evaluations. Examples meant to teach tool use can also limit exploration once the model understands the tools.
After a model upgrade:
- retest the old planning rules and reminders, not only the model;
- drop tool-use examples the model no longer needs;
- keep what does not age: clear tool interfaces, reliable state, observable actions and a controlled execution environment.
06 · Security is a property of the whole system
A model reaches files, tools, memory and permissions through its harness. HarnessSafe placed attacks in agent stores across 328 cases and scored how early each was contained. The same GPT-5.6 Sol scored 62.3 in Codex CLI and 39.4 in Claude Code.
Source: State of AI Report 2026, slide 202, CC BY 4.0.
Approval layers help, and their limits need measuring. Anthropic’s command-approval classifier blocked 89% of dangerous commands inserted into test sessions, against 13.6% for humans; in a separate evaluation, 17% of real overeager actions still got through. Installed skills are part of the attack surface too: a study the report cites found vulnerabilities in 26.1% of 31,132 marketplace skills.
What to do: test the model, the harness and the approval system together, and review installed skills as third-party code. Our registry of 15 prompt injection cases shows the same pattern in shipped products.
07 · AI-native companies grow about three times as fast
In Standard Metrics’ preliminary Q2 data, companies where AI is the product grew 256% a year at the 75th percentile in the $1–20M ARR band, against 90% for companies that added AI to existing software. Above $20M the comparison was 172% against 53%. The data shows an association; faster growers may simply spend more on AI.
The revenue at the top moved fast as well. OpenAI and Anthropic together reached about $105B in annualised run-rate by late summer, up from about $30B at the start of the year. Spending is concentrated: in Ramp’s August data the median firm in the top 1% of AI spenders paid $7,205 per employee per month, against $12.50 for the median firm (business section of the report).
08 · Domain data takes open models past the frontier on their own work
Application companies can start on frontier APIs and then use expert feedback to post-train a model for their domain. Harvey’s post-trained GLM-5.2 ran in production at 54.8% lower cost per review-table cell than Sonnet 5, and scored 0.903 on Harvey’s answer-quality measure against 0.867 for Fable 5. Mercor trained an open Qwen model on 1,928 expert tasks and lifted its first-attempt pass rate on 480 held-out tasks from 16.11% to 27.29%.
Source: State of AI Report 2026, slide 113, CC BY 4.0. Results are company-reported on specialist evaluations.
What to do: before post-training, collect what a correct answer looks like in your customers’ work and the data to test it. Taking control of the model pays off when the gain in quality, latency or cost covers the work.
09 · Agents that learn on the job: the improvement loop
One of the report’s nine predictions for the next twelve months: an agent halves its failure rate on new tasks after a month of customer work, without a model upgrade. The authors grade themselves each year; last year’s ten predictions produced two hits, five partial outcomes and three misses.
The production loop the report describes is the way to get there:
- Record every task, tool call, outcome and correction.
- Turn repeated failures into tests you rerun.
- Decide what to change: context, memory, routing, a tool or the model.
- Count correctly completed jobs and ignore partial credit.
Collecting transcripts without evaluating what worked does not count as learning.
10 · What the report does not prove
- Simulations are not markets. KellyBench and E-CommerceBench measure risk handling and learning under their own rules. They say nothing about business profit.
- Growth data is preliminary. The Standard Metrics comparison shows an association. It does not show cause.
- Company numbers are self-reported. Harvey, Mercor and the frontier labs report their own results on their own evaluations.
- Benchmarks age fast. The research section notes that Epoch AI found substantive flaws in nine of its first 15 benchmark reviews.
FAQ
What is the State of AI Report 2026? An annual open-access report by Nathan Benaich and Air Street Capital on AI research, industry, compute, safety and policy. The 2026 edition was published on 8 October 2026 under a CC BY 4.0 licence.
Can an AI agent improve without changing the model? Yes. In a controlled coding study cited by the report, GLM-5.1 with fixed weights rose from 52.5% to 65.5% on 100 SWE-bench Verified tasks after its tools, context and feedback changed.
Why do high benchmark scores not mean the task is done? Partial credit counts individual requirements. GPT-5.6 Sol met 87.9 out of 100 on requirements across 97 scientific workflows but fully completed 20.6% of them.
Does the same model have the same security in every agent? No. On HarnessSafe the same GPT-5.6 Sol scored 62.3 in Codex CLI and 39.4 in Claude Code.
How should a team improve an agent in production? Record tasks, tool calls, outcomes and corrections; turn repeated failures into tests; decide what to change; and count correctly completed jobs.
