Failure taxonomy
Finding. Every one of the 2,500 attempts in the series ran to completion: no cap bound, no harness crashed, no grade was invalid. The only non-verdict outcomes were 14 empty patches, and those were the model's decisions, not the scaffold's.
"Unresolved" is not one thing. The framework records how every attempt ended and what category of outcome it was, and keeps the framework's own failures (crashes, caps) apart from the agent's.
How attempts ended
terminated_reason per harness — the same 500 instances each
An attempt that 'completed' is one where the agent itself stopped; the verdict (resolved / unresolved) is a separate axis, shown in the verdict table. Caps and crashes are the framework's limits or defects, and are reported as such rather than folded into 'unresolved'.
Categories
| category | meaning | counted in gradeable? |
|---|---|---|
| resolved | the patch applied and the fail-to-pass tests pass, with no pass-to-pass regression | yes |
| unresolved | the patch applied but the tests did not all pass | yes |
| empty patch | the agent finished without changing the repository | yes — an empty patch is a real outcome |
| harness crash | the harness process failed for a reason the framework owns | no — excluded from the denominator and reported |
| budget / turn / token cap | the run's enforced limit ended the attempt | yes — the limit is part of the experiment |
| grade invalid | the evaluation itself could not produce a verdict | no |
What the series showed
- Every one of the 2,500 attempts ended as
completed— no cap bound, no harness crashed, no grade was invalid — so the chart above is five full bars and the per-attempted and per-gradeable denominators coincide. That is the point of the chart: it shows the absence, with its n. - Empty patches were rare (14 across the five runs: 5 OpenCode, 3 Claude Code, 3 mini-swe-agent,
2 custom minimal, 1 Codex CLI) and, on inspection, model-side: the model decided the issue was
already fixed, reverted its own fix, or collapsed its tool-call format. Only
django__django-13513is an empty patch in all five runs;sphinx-doc__sphinx-9229andpylint-dev__pylint-7080are empty only in the OpenCode run and plain unresolved in the other four. - No attempt hit the per-instance budget cap in this series; the caps were set from the calibration runs so that they bind on a runaway, not on a normal attempt.
- One failure mode the export cannot see: OpenCode's permission system ended 23 sessions on its
own (see Harness adapters); those attempts are
completedhere because the harness exited cleanly.
- run
- 01789087148400722169-0c9303b3 · 2026-09-11
- model
- openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-opencode)
- harness
- OpenCode · CLI 1.18.18 (pinned in the harness image build at this framework commit)
- framework
- 631be843b55ccabe33347f0a7de2f7ee0676ce1a
- harness image
- sha256:c8c56129b240e898fb5de452b50e36ad1aedf04fada946592ee98ca757753564
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs