Head-to-head
Finding. The five harnesses solve largely the same problems and fail the same ones; the instances no harness resolves are the model's hard core, and the best-harness-per-instance oracle adds only a few points.
Aggregate rates hide which problems each harness solves. On the same model and the same 500 instances, the sets overlap heavily but not completely.
| set | instances |
|---|---|
| resolved by every harness | 327 |
| resolved by at least one harness (oracle over harnesses) | 425 |
| resolved by none | 75 |
| resolved only by OpenCode | 5 |
| resolved only by custom minimal agent | 5 |
| resolved only by Claude Code | 7 |
| resolved only by Codex CLI | 2 |
| resolved only by mini-swe-agent | 2 |
Instances no harness resolved (75)
astropy__astropy-13033, astropy__astropy-13236, astropy__astropy-13398, astropy__astropy-13977, astropy__astropy-14598, django__django-10554, django__django-10999, django__django-11087, django__django-11141, django__django-11400, django__django-11532, django__django-11734, django__django-11820, django__django-11964, django__django-12308, django__django-12406, django__django-13195, django__django-13212, django__django-13513, django__django-13794, django__django-14011, django__django-14034, django__django-14140, django__django-14155, django__django-14170, django__django-14315, django__django-14792, django__django-15098, django__django-15252, django__django-15629, django__django-15732, django__django-15916, django__django-15973, django__django-16256, django__django-16263, django__django-16502, django__django-16631, django__django-16667, matplotlib__matplotlib-21568, matplotlib__matplotlib-23299, matplotlib__matplotlib-23476, matplotlib__matplotlib-24870, matplotlib__matplotlib-25479, matplotlib__matplotlib-26208, matplotlib__matplotlib-26466, mwaskom__seaborn-3187, pydata__xarray-6599, pydata__xarray-6992, pydata__xarray-7229, pylint-dev__pylint-4551, pylint-dev__pylint-4604, pylint-dev__pylint-4661, pylint-dev__pylint-7080, pytest-dev__pytest-5840, scikit-learn__scikit-learn-25747, scikit-learn__scikit-learn-26194, sphinx-doc__sphinx-10614, sphinx-doc__sphinx-11510, sphinx-doc__sphinx-7462, sphinx-doc__sphinx-7590, sphinx-doc__sphinx-7748, sphinx-doc__sphinx-9229, sphinx-doc__sphinx-9461, sympy__sympy-13031, sympy__sympy-13798, sympy__sympy-13974, sympy__sympy-16597, sympy__sympy-17630, sympy__sympy-18199, sympy__sympy-18763, sympy__sympy-20428, sympy__sympy-20438, sympy__sympy-21596, sympy__sympy-21930, sympy__sympy-22080
Pairwise agreement
| OpenCode | custom minimal agent | Claude Code | Codex CLI | mini-swe-agent | |
|---|---|---|---|---|---|
| OpenCode | 385 | 361 | 361 | 366 | 359 |
| custom minimal agent | 361 | 384 | 358 | 360 | 359 |
| Claude Code | 361 | 358 | 384 | 360 | 355 |
| Codex CLI | 366 | 360 | 360 | 383 | 358 |
| mini-swe-agent | 359 | 359 | 355 | 358 | 379 |
Each cell is the number of instances both harnesses resolved; the diagonal is the harness's own total.
Reading it. The oracle-over-harnesses number is the resolve rate you would get by picking, per
instance, the best of five scaffolds — an upper bound on what the scaffold can contribute with this
model. The instances no harness resolved are the model's (or the benchmark's) hard core;
pylint-dev__pylint-4661, for example, is unresolved in every run here because its test patch
imports a package (appdirs) that the gold patch adds as a dependency — the gold patch passes
validation, an agent patch that fixes the bug without adding the dependency cannot.
- run
- 01789087148400722169-0c9303b3 · 2026-09-11
- model
- openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-opencode)
- harness
- OpenCode · CLI 1.18.18 (pinned in the harness image build at this framework commit)
- framework
- 631be843b55ccabe33347f0a7de2f7ee0676ce1a
- harness image
- sha256:c8c56129b240e898fb5de452b50e36ad1aedf04fada946592ee98ca757753564
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs