Versus the official leaderboard
Most of what this site measures has no published counterpart: nobody else runs five harnesses on one model under one set of limits, and the harness vendors do not publish MiniMax numbers. One cell does have a counterpart. The official SWE-bench leaderboard's "Bash Only" track is exactly mini-swe-agent, the reference agent, and MiniMax submitted M2.5 to it in February 2026. Our mini-swe-agent × MiniMax M2.5 run on the same 500 Verified instances is therefore checkable.
The comparison
| official leaderboard entry | this rig | |
|---|---|---|
| entry | MiniMax M2.5 (high), 20260217_mini-v2.0.0_minimax-2-5-high | run 01788723779415700648-b4338f67 |
| harness | mini-swe-agent 2.0.0 | mini-swe-agent 2.4.6 |
| model | minimax-m2.5, "high" reasoning effort | minimax-m2.5 through OpenRouter, pinned to MiniMax's own endpoint; reasoning effort high, temperature 1.0, top_p 0.95, set on the gateway alias |
| date | submitted 2026-02-17 (model released 2026-02-12) | 2026-09-06 |
| resolved | 379 of 500 · 75.8% | 379 of 500 · 75.8% (Wilson 95% 71.9–79.4) |
| total inference cost | $36.64 | $32.72 |
| cost per instance, mean (median) | $0.073 ($0.049) | $0.065 ($0.040) |
| API calls per instance, mean (median) | 60.5 (52) | 53.1 (42.5) |
| checked by the leaderboard maintainers | not yet | — |
Instance by instance: 350 resolved by both, 92 by neither, 29 resolved only by the leaderboard entry, 29 only by this rig — 88.4% agreement, and a swap of exactly 29 for 29. The disagreement is the size of run-to-run variance for this model and harness, and it is symmetric.
On the 78-instance subset used for the model experiments, the leaderboard entry resolved 56 of 78 at $4.56 and this rig's mini-swe-agent × MiniMax run resolved 58 of 78 at $4.37.
For context, the previous generation on the same track: MiniMax M2 with mini-swe-agent 1.17.0, November 2025, resolved 305 of 500 (61.0%) at $214 — a fifth the cost per resolve a generation later.
What this does and does not establish
- It establishes that the rig grades the way the official leaderboard grades. Same dataset revision, same official images, same harness family, same model; the aggregate matches exactly, the per-instance sets overlap 88%, and the cost is within 11%. That is the calibration point for every other number on this site, which is why this page exists.
- It does not make the two runs identical. The leaderboard run used mini-swe-agent 2.0.0 and the model's "high" reasoning setting; this rig used 2.4.6 with the default setting as served through OpenRouter, under its own limits (350 turns, $2 per instance, a 262k context declared to the harness). Twenty-nine instances flipped each way; on 500 instances that is inside the noise either of us would see re-running our own submission.
- The other four harnesses have no leaderboard counterpart. OpenCode, Claude Code, Codex CLI and the minimal agent are compared only against each other, on this site.
Sources
The leaderboard entry's metadata.yaml and per_instance_details.json are copied into this
site's data directory from the SWE-bench/experiments repository
(evaluation/verified/20260217_mini-v2.0.0_minimax-2-5-high/), so the comparison can be
re-derived from committed files; the per-instance agreement above is computed from those and from
this rig's run export.
- run
- 01788723779415700648-b4338f67 · 2026-09-06
- model
- openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-mini)
- harness
- mini-swe-agent · CLI 2.4.6 (pinned in the harness image build at this framework commit)
- framework
- ff0c00d8838fa75b3164d8d21bf91cc3e188b2ee
- harness image
- sha256:b294634551e3cf677e76fa011fabc28272eca1ed13d8d4a930d7b89270229374
- gateway config
- bb22fcd7bad15793e5814f4abbd4dfba760ea3c76a3d8a69d7ec935a3ad2df7e
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- limits
- max_tokens=not set · max_cost_usd_per_instance=2 · attempts_per_instance=1 · temperature=1