How Open-Source Agents Matched Frontier AI at 1/30th the Cost

Over three weeks, 147 builders took on Grounded Reasoning on Databricks' OfficeQA benchmark. Top teams averaged ~70% accuracy at roughly $1.74 per run — near-frontier performance at 1/30th the cost of closed-source baselines.

Sentient Arena Cohort 0 Winners

How Open-Source Agents Matched Frontier AI at 1/30th the Cost

Over three weeks, 147 builders drawn from 1,200+ applicants took on Grounded Reasoning: the inaugural Arena challenge, built on Databricks' OfficeQA benchmark.

Every team had to use the same open-source model (MiniMax M2.5) and build an open-source coding agent to beat the benchmark’s oracle setup.

The headline result: Top six teams averaged ~70% accuracy on OfficeQA at roughly $1.74 per run. That’s near-frontier performance at 1/30th the cost of closed-source baselines. While closed-source models may still return the best absolute performance, good engineering with open-source models can get you more than 10× the performance-per-dollar.

Breakdown of Results

Cohort 0 builders were tested against two models: the open-source MiniMax M2.5 and the closed-source Opus 4.5 (Anthropic). The only levers were prompt engineering, skill files, configuration, and harness choice across goose, OpenHands, OpenCode, and OpenAI’s Codex CLI.

The top six teams averaged ~70% accuracy with MiniMax M2.5 and Goose at an average of only $1.74 per run. Holding everything else constant (prompts, skills, and configuration), we reran the agents with Opus 4.5, seeing a moderate increase in performance but a substantial ~30× jump in cost.

The leaderboard compares the best runs on MiniMax M2.5 versus Opus 4.5 for the top six teams. Closed-source wins on accuracy (~80% vs ~70%), but MiniMax M2.5 ran at an average of $1.74 per run versus $56.53 for Opus 4.5. That’s roughly 30× cheaper. When you factor cost into the final score (score decreases with cost), MiniMax M2.5 comes out ahead for every team in the top six.

The takeaway: Open-source models aren't just cheaper. With the right harness and prompting, they hold their own against closed-source competitors on accuracy-per-dollar.

What We Learned

1. There is still an LLM capability ceiling.

No matter how clever the prompts or skills, teams kept hitting the same ceiling. With MiniMax M2.5, Deepmind Ace (Pranav Patel) broke the questions into three groups: about 45% of the time the model always gets it right, 25% it always gets wrong, and 30% it flips on randomly. The math roughly works out to ~60–68% of expected accuracy, which is exactly where most teams landed. The team Zero Node (Jwalin Shah, Tarun Sivaramakrishnan & Sanjay Subramanian) independently confirmed the same ceiling.

The reason it gets stuck: the model finds the right document but then reads the wrong number off the table. 68% of wrong answers are "right file, wrong cell." That's a limit of this specific model, not a prompting problem. This ceiling naturally increased when swapping to Opus 4.5, a smarter model, and after investigating the frequency of the aforementioned retrieval errors, we found naturally less of them, while the majority of the “always wrong” questions remained unsolved.

The takeaway: with all the prompting, skills, and tools necessary, even the best LLMs still struggle on more complex multi-source reasoning tasks like OfficeQA, and trace data from environments like Sentient Arena will be instrumental in training the next frontier of LLMs that hopefully make less of these mistakes.

2. Small setup choices had huge effects.

Switching to the goose harness beat alternatives by ~10% in accuracy and was 8× cheaper than the next option. We decided to further investigate this with independent testing. On Terminal Bench 2.0, we found a similar story — Goose was 20× more token efficient (and cheaper) than OpenHands, and more than 40× cheaper than OpenCode.

Denser prompts beat longer ones. Team-tiok (Vikranth Reddimasu) and Dolores Research (Leon Liu) found that prompt density enabled lower powered models like MiniMax M2.5 better than long prompts. By carefully designing short and concise prompts, they were able to hit the same accuracy as verbose prompts while cutting cost by 87%.
One of the most common failure modes was the model continuing to search or reason after it had already found the answer. Dolores Research’s (Leon Liu) fix was a simple rule: telling the model "write your answer now" before every step. That single instruction eliminated a whole class of failures worth 22 questions.

3. The unexpected findings — how you talk to the model matters a lot.

Cat Varnell tested the same prompts under different tonal framings. On select query subsets, prompting the model as if it were a tool yielded scores of 25%. A warmer framing, like "notes your colleagues left for you", scored 88%. Same content, different relationship, 3× the accuracy.
Zero Node (Jwalin Shah, Tarun Sivaramakrishnan & Sanjay Subramanian) found that telling the model it would lose points for skipping a question (which is a lie since there's no penalty) cut skipped questions by 65%, from 49 down to 17. The scoring formula has no wrong-answer penalty, so a guess is strictly better than a blank: some guesses land, and the cost of writing one is negligible. The lie just nudged the model to take the free expected value it was leaving on the table.

Bottom line

Where this leaves us: On pure accuracy, closed-source wins. On accuracy-per-dollar, an open-source model like MiniMax M2.5 can win by a lot.

Thank you to the builders. Everything we learned in Cohort 0 came from builders willing to share in the open. That's the real argument for open-source AI. Thank you to every Arena team who showed their work.

What's next

Cohort 0’s submissions generated 108,000+ reasoning traces — recordings of how the AI agents thought through each problem. We will soon be releasing a clean version of this dataset on Hugging Face, for anyone interested in failure analysis, evolving your own agents, or fine-tuning LLMs.

Think you can break the LLM capability ceiling?

Pre-register for the public Arena: Challenge 0 → sentient.xyz/arena

Missed Cohort 0? Here's everything you need to catch up: