What the runs showed
Six findings. Each card links to the page with the run ids, intervals and provenance behind it.
The reference run reproduces the official leaderboard
mini-swe-agent × MiniMax M2.5 resolves 379 of 500 here and 379 of 500 on swebench.com's Bash Only entry, with 88.4% per-instance agreement and cost within 10.7%. Same grading, same images, independently run.
Versus the official leaderboard →The harness decides what gets burnt, not what gets solved
Five harnesses on one model and the same 500 problems land between 75.8% and 77.0% with overlapping intervals. The bill differs 1.5×, and the judge's redundant-call flags run from 52 to 258 of 500 attempts depending on the scaffold.
By harness →Cheaper per token is not cheaper per task
Laguna xs 2.1 and MiniMax M2.5 on the same harness, prompt, tools and limits resolve the same share of 78 problems; Laguna carries 7.5× the prompt tokens and costs 4.5× as much per resolved instance. Only the per-call ledger shows why.
Model vs model →Efficiency can be engineered, and it has to be enforced
Five working rules in the system prompt cut the bill by a quarter to a third on the same 78 problems with no measurable change in outcome. The rule both models ignored, stop once tests are green, is the one the harness now has to enforce; the judge measures the waste that remains.
What a prompt can buy →A two-tool loop keeps up with the commercial scaffolds
The framework's own control agent, one loop and two tools, resolves 384 of 500 at $34: inside the noise of the four CLIs. What a scaffold adds shows up in the bill and in the judge's efficiency findings, not in the resolve rate.
The custom minimal agent →The judge found the rig's bug first, and twelve memorisation flags in 2,509 attempts
Rubric v3 read every attempt of the five runs. Its first pass flagged 221 of 500 OpenCode attempts for a search tool broken in the isolated image, a framework defect fixed and re-run. Across all five it raised 12 contamination flags, none citing anything reachable inside the container, and one class of test gaming no static check sees.
LLM judge →How a run works
From a launch request to a graded, judged, published result — every step enforced by a component that does not trust the agent.
Launch
Pick the harness, the model, the instances and the limits. A per-run provider key carries the budget as a hard cap.
Isolated agents
One container per instance, no internet, repository history pruned at the base commit. A local proxy shim meters every call.
Paced inference
One gateway for every call. A token-bucket pacer and an autoscaler keep hundreds of agents inside the provider's real limits.
Official grading
The official SWE-bench harness in the official per-instance images, pulled by digest, each one gold-patch validated first.
Judge and publish
An evidence-required LLM judge reads the trajectories; every number ships with its run id, model, commit and limits.
What it measures, and what it refuses to hide
Resolve rate is one number. The rig records the cost, the behaviour and the integrity checks behind it.
Any harness × any model
OpenCode, Claude Code, Codex CLI, mini-swe-agent and a minimal control agent behind one contract; any OpenRouter model through one alias.
Harness adapters →Limits enforced, not hoped for
Turns, tokens, dollars and wall-clock are capped by the shim and the gateway, outside the harness. Every cap that binds is reported by name.
Limits enforced →Integrity before results
No egress, pruned history, gold-patch validation, leak detection, both denominators — and the story of the leak the rig caught in itself.
Integrity →Cost is a first-class result
Metered per call from the provider's own usage. Cost per resolved instance separates harnesses that resolve rate cannot.
Cost →A judge that has to cite
Nine rubric dimensions; a flag without cited evidence is demoted, never kept. It found the framework's own defects first.
LLM judge →Full artifacts, frozen
Predictions in the official format, trajectories, eval reports, per-call ledgers and timeline bundles with hashes — everything on this site is a committed file.
Downloads →The headline number
Five harnesses on the same model and the same 500 problems. The scaffold barely moves the resolve rate; it moves the bill.
Cost per resolved instance
USD of inference per resolved SWE-bench Verified instance · minimax/minimax-m2.5 · lower is better
Inference spend only, from the gateway's own metering (compute cost is reported separately on the Cost page). Sorted cheapest first. A missing bar is an unmeasured run, not a free one.
How to read it
- Resolve rates run from 75.8% to 77.0% and every 95% interval overlaps every other — at n = 500, the harness does not change the outcome by more than the noise.
- Cost does separate them: the cheapest harness spent about two thirds of what the most expensive one did for the same outcome.
- The OpenCode row is a re-run on a fixed image; the first run, whose search tools were broken, is kept as a documented defect on the Integrity page.
- run
- 01789087148400722169-0c9303b3 · 2026-09-11
- model
- openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-opencode)
- harness
- OpenCode · CLI 1.18.18 (pinned in the harness image build at this framework commit)
- framework
- 631be843b55ccabe33347f0a7de2f7ee0676ce1a
- harness image
- sha256:c8c56129b240e898fb5de452b50e36ad1aedf04fada946592ee98ca757753564
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
Built to be argued with
Every claim on this site traces to a run id, a commit and a committed export. Anything not measured says so — it is never rendered as zero.
Reproduce it- Both denominators and a Wilson 95% interval on every rate
- Provenance block on every page that shows a number
- Official grading, official images, dataset revision pinned — reproduces the official leaderboard's mini-swe-agent × MiniMax M2.5 entry to the decimal
- Per-call ledgers and trajectories published per run
- One person, four weeks, and the receipts for what it cost