Headline results
Finding. The scaffold does not move the resolve rate on this model; it moves the bill by half. Cost per resolved instance separates the five harnesses, resolve rate does not.
Five agent harnesses, the same model (MiniMax M2.5), the same 500 SWE-bench Verified instances, the same per-instance budget, the same isolated environments. From dispatch to the last verdict, the OpenCode re-run took 1.5 hours (peak 63 concurrent agent tasks, 36 grading at once) and the Codex run 2.9 hours (peak 77); every run's fleet timeline is in the Explorer.
Cost per resolved instance
USD of inference per resolved SWE-bench Verified instance · minimax/minimax-m2.5 · lower is better
Inference spend only, from the gateway's own metering (compute cost is reported separately on the Cost page). Sorted cheapest first. A missing bar is an unmeasured run, not a free one.
Resolve rate by harness, with 95% intervals
minimax/minimax-m2.5 · SWE-bench Verified · n = 500 per harness · both denominators, always
Error bars are Wilson 95% intervals. Where intervals overlap, the harnesses are not distinguishable at this sample size — a two-point gap is not a result. 'per attempted' counts every instance the run launched; 'per gradeable' excludes infrastructure failures (ADR-0038). Here the two coincide because no attempt was excluded.
The table behind the charts
| harness | attempted | gradeable | resolved | rate / attempted | rate / gradeable | cost | $ / resolved | tokens in / out |
|---|---|---|---|---|---|---|---|---|
| OpenCode 0c9303b3 | 500 | 500 | 385 | 77.0% [73.1, 80.5] | 77.0% [73.1, 80.5] | $26.29 | $0.068 | 534.1M / 4.4M |
| custom minimal agent 05bd05b8 | 500 | 500 | 384 | 76.8% [72.9, 80.3] | 76.8% [72.9, 80.3] | $33.98 | $0.088 | 683.1M / 5.8M |
| Claude Code 1f722383 | 500 | 500 | 384 | 76.8% [72.9, 80.3] | 76.8% [72.9, 80.3] | $39.50 | $0.103 | 36.7M / 6.0M |
| Codex CLI c10df654 | 500 | 500 | 383 | 76.6% [72.7, 80.1] | 76.6% [72.7, 80.1] | $30.63 | $0.080 | 635.6M / 4.7M |
| mini-swe-agent b4338f67 | 500 | 500 | 379 | 75.8% [71.9, 79.3] | 75.8% [71.9, 79.3] | $32.72 | $0.086 | 616.9M / 5.8M |
How to read this
- The five resolve rates are statistically indistinguishable. Every interval overlaps every other. On this model, at n = 500, the scaffold does not move the resolve rate by more than the noise. That is itself the finding.
- Cost is not indistinguishable. The cheapest harness spent about two thirds of what the most expensive one did for the same outcome. Cost per resolved instance separates them; resolve rate does not.
- One row can be checked against the world. The mini-swe-agent run resolves 379 of 500 — 75.8%, the same figure as the official SWE-bench "Bash Only" leaderboard entry for MiniMax M2.5, with 88% per-instance agreement and cost within 11%. See Versus the official leaderboard.
- One harness was re-run. The first OpenCode 500-run used an image on which OpenCode's grep
and glob tools were broken — the judge flagged 317 of 501 attempts for an environment problem,
262 citing
ripgrep— and its permission system ended 23 sessions on its own. Both were fixed in the image and the full split was re-run on 2026-09-11; that re-run is the OpenCode row in this table (385 of 500, $26.29, against 388 and $28.99 on the broken image). The original run stays in the Explorer and on the Integrity page as the defect the judge found.
- run
- 01789087148400722169-0c9303b3 · 2026-09-11
- model
- openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-opencode)
- harness
- OpenCode · CLI 1.18.18 (pinned in the harness image build at this framework commit)
- framework
- 631be843b55ccabe33347f0a7de2f7ee0676ce1a
- harness image
- sha256:c8c56129b240e898fb5de452b50e36ad1aedf04fada946592ee98ca757753564
- gateway config
- bb22fcd7bad15793e5814f4abbd4dfba760ea3c76a3d8a69d7ec935a3ad2df7e
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- limits
- max_tokens=not set · max_cost_usd_per_instance=2 · attempts_per_instance=1 · temperature=1
The provenance block above is the OpenCode run's; the other four runs share the framework commit, dataset revision and network posture and differ in harness and harness image. The full block for each run is on By harness.