Inside 13,000 Agent Runs: Why AI Agents Succeed or Fail on OfficeQA

The OfficeQA Public Challenge generated more than 73,000 agent runs across 782 scored submissions. We pulled the complete step-by-step trajectories from a representative 20% sample — 13,483 individual runs across 156 submissions and all four harnesses: Goose, OpenAI's Codex, OpenHands, and OpenCode — to find out what separates a run that gets the right answer from one that doesn't.

TL;DR

Agent success comes down to how efficiently the agent works at a task, and what kind of task it is:

• Whether a task gets solved is mostly about the task, not the agent. The four harnesses agree roughly 90% of the time on which questions are easy and which are hard.
• When an agent is heading toward a wrong answer, it takes 50% more steps getting there, and it burns more tokens and money along the way.
• Errors along the way don't predict failure. Passing and failing runs hit errors at almost identical rates.
• Tool variety helps, but not as much as you'd think.

What we analyzed

In OfficeQA, every agent had to answer 94 hard numerical questions about a 697-document U.S. Treasury Bulletin corpus (e.g. "What was the total public debt outstanding in June 1974?") — fully offline, no internet, everything enforced server-side on the same model. Participants competed on one of four open-source harnesses: Goose, Codex, OpenHands, and OpenCode.

For this study we sampled 156 submissions (20% of the scored population), balanced across all four harnesses and across high, mid, and low-scoring teams, and extracted the full trajectory of every run to understand which tools the agent called, what it read, what it computed, and whether it landed the answer. That's 13,483 complete episodes.

Finding 1: Difficulty lives in the task, not the harness

For each of the 94 questions, we measured how often agents got it right and whether different harnesses agreed on which ones were harder to get right.

The questions split into clear bands: 36% are usually solved, 36% are very hard, 26% are genuinely contested, and a couple are effectively impossible for everyone.

Task difficulty distribution across the 94 OfficeQA questions.

Across harness pairs, agreement on pass/fail per task runs 88–93%. So agents do agree — and overwhelmingly so.

Cross-harness agreement on which tasks pass or fail sits between 88% and 93% for every pairing.

In plain terms, if a question is hard, it's hard for everybody. A confusing table or a tricky unit conversion is a property of the question, not of your harness choice. Only the ~26% contested questions are where the agent's strategy decides the outcome.

The next hypothesis we tested: do the harnesses differ, and can any of them uniquely solve tasks the others can't?

We ran them through three criteria:

1. Speed. On typical runs, the harnesses are similar (~4 min median), but the tails differ sharply. Codex is by far the slowest, with a 95th-percentile run time of ~36 minutes (and a long tail past an hour), versus ~15–17 min for OpenHands and OpenCode. Goose's traces don't record per-step timestamps, so Goose can't be timed on this criterion.
2. Who handles the hardest tasks. No harness is dramatically better at the hard stuff. On the 36 hardest tasks (those solved <35% of the time overall), the harnesses are competitively close, with Goose modestly ahead.
3. Capacity is shared, not exclusive. No single harness was able to solve certain tasks alone. 37 of 94 tasks were solved by all four harnesses, and only 1 task was solved by a single harness (OpenHands). The remaining hard tasks tend to defeat everyone: 42 tasks weren't reliably solved by any harness — which goes back to our finding that the difficulty lies in the task, not the harness.

Finding 2: Failing runs take substantially more steps, costing more

Across every harness, failing runs take substantially more steps than passing ones.

Failing episodes take significantly more steps — from +11% on Codex to +50% on Goose.

And it isn't just steps. Failing runs burn more tokens, and on the one harness with per-run cost data, a wrong answer averaged $0.120 versus $0.086 for a right one — roughly 40% more expensive per failure.

Failing episodes consume more steps, tokens, and cost.

When an agent is heading toward a wrong answer, it doesn't fail quickly — it thrashes, taking many more actions while getting nowhere. That's a useful early-warning sign: a run that's ballooning in step count is more likely to be failing. And it isn't just wasted time — failure literally costs more compute and money per episode.

Finding 3: Errors along the way are NOT a warning sign

Counterintuitively, we found that runs which hit more errors — "file not found," failed commands, empty search results — don't fail more often. At any given point in a run, passing and failing episodes show almost identical error rates. For example, ~5.5% vs ~5.3% at step 5. The two lines sit right on top of each other.

Error rate as a run progresses. The "passing" and "failing" lines sit right on top of each other — errors happen at the same rate whether or not the agent ends up correct.

Hitting errors is just a normal part of how these agents work: they probe, get an error, adjust, and move on. An error is not a sign the run is doomed. So "counting the errors" is a bad way to predict failure, even though intuitively it feels like it should work.

Finding 4: Trying different tools helps, but only a little

Tool diversity is defined as how many different tools an agent used in a run, measured as entropy: a higher number means the agent mixed up its tools, a lower number means it repeated the same tool.

We wanted to test whether tool diversity correlates with agent success. What we found is that across all four harnesses, successful runs used slightly more varied tools than failing ones — the signal points the right way, but it's weak (statistically, only a little better than a coin flip at predicting failure).

So an agent that keeps hammering the same tool over and over is a mild sign of being stuck, but "use more tools" is not a magic recipe for success.

Finding 5: Repetitive tool calls look like getting stuck

We measured how often an agent calls the same tool twice in a row (a "retry") and whether that correlates with failure.

Runs that struggle tend to retry the same action repeatedly instead of trying something different. Taken together with Finding 2 (failing runs take more steps and cost more) and Finding 4 (failing runs show less tool variety), a consistent picture emerges: a failing run isn't one that hits an error and stops — it's one that gets stuck in a loop, repeating itself over many steps without making progress.

Harness choice: every harness has a fingerprint, but no single winning style

Beyond pass/fail, each harness reaches for tools in a recognizably different way:

• Codex is the most shell-centric. Nearly everything runs through one command-line tool — ~20 shell calls per run — and it rarely uses dedicated search or read tools.
• OpenCode is the most tool-diverse. It leans into dedicated search and reads far more than the raw shell. This matches its high tool-variety score from Finding 4.
• Goose and OpenHands sit in between, shell-heavy with a real search/edit mix.

The heatmap shows the average number of calls to each type of tool per run. We group the many individual tools into families — shell, search, read — so the four frameworks can be compared fairly.

How each harness succeeds

Most of this piece focused on failures. Flipping it around to look only at the successful runs, the harnesses win in noticeably different ways.

Goose's traces carry no timestamps.

How they win:

• OpenCode wins "surgically" — the fewest steps (18 median), the most varied toolkit, rarely repeating itself.
• Codex wins by "brute force." It routes almost everything through a single shell tool (near-zero tool diversity), repeats actions often, and takes the most steps. It gets there, just less elegantly.
• OpenHands and Goose sit in between. Both get correct answers; they just look completely different on the way there.

Speed follows the same pattern. Typical winning runs are quick on every harness (~3 min median), but the tails diverge sharply: Codex's 95th-percentile winning run stretches to ~23 minutes. There's no single harness or recipe for success. A lean, varied approach (OpenCode) and a repetitive, shell-only grind (Codex) both get correct answers — they just look completely different along the way.

Rounding up: the signals that predict agent failure

We scored every signal on how well it can tell a failing run apart from a passing one, per harness. The scale is simple: 0.50 = useless, and the higher the score, the better.

The single most reliable tell is just how long the agent keeps working — the flailing signal from Finding 2. The obvious candidate, error rate, is essentially a coin flip. This confirms the pattern: watch the agent's effort, not its errors.

Conclusions

Put it together and the story is simple:

1. Solvable-or-not is mostly the question's call. Agents largely agree on what's hard.
2. Winning agents fail fast and cheaply. Losing agents flail — more steps, more repetition, no progress.
3. Don't trust the obvious signals. Error messages don't predict failure. The strongest tell is simply how long the agent thrashes.

For anyone building agents, the actionable lesson is to detect the flailing early and cut it off. A run that's piling up steps and repeating itself is the one to intervene on — far more than one that just hit an error.

What's next

These 13,483 trajectories are a preview of what the full OfficeQA trace dataset makes possible: failure analysis, self-correcting agents that notice when they're stuck, and training data for models that thrash less.

• Check out the Sentient Arena website
• Explore the final OfficeQA leaderboard
• Read our Meet the Winners of the OfficeQA Public Challenge article
• Catch up with our Arena YouTube playlist