Harness adapters
What it is. One interface (run(instance, model_alias, limits) → patch, trajectory, usage)
implemented for five harnesses: Claude Code, Codex CLI, OpenCode, mini-swe-agent (SWE-bench's
reference agent) and a purpose-built minimal agent. Any harness pairs with any model through one
gateway alias; the framework treats every harness as a black box that must produce a patch.
What every adapter must do the same way
| concern | how |
|---|---|
| repository | run SWE-bench's own install_repo_script into /testbed — never the framework's own clone, which once leaked the gold patch through git history |
| model | the harness's native provider config is pointed at the gateway alias; the API key is the run's virtual key |
| budget | tokens, turns, cost and wall-clock caps are enforced outside the harness (below), not by trusting its own counters |
| output | the diff of /testbed against the base commit, the harness's native trajectory, and a normalised trajectory the judge can read |
| network | none; the task's subnets have no default route, and the only reachable endpoints are the inference gateway and the in-VPC AWS endpoints for artifacts, queues and live progress (ADR-0028, ADR-0033); the repository is already in the image, so nothing is cloned at run time |
The local proxy shim (ADR-0019)
Four of the five harnesses cannot enforce a spend cap. Polling the gateway's spend log was rejected (lag, and a race with the last call). Instead every task runs a local proxy on loopback; the harness talks to it, it talks to the gateway. The shim counts tokens on the way through, holds a call when the pool's pacer says so, and ends the session when the instance's budget is spent. It is also where the pacer's weighted charge is applied (cached prompt tokens cost the pool less than uncached ones).
Per-harness notes that matter for reading results
- OpenCode: its grep and glob tools download
ripgrepat first use; with no network they failed silently on every run until the judge flagged it. Fixed by a static binary on the image. Its permission system also auto-rejected tool calls outside/testbedand ended the session until an explicit permission block was added. In the published run that happened on 23 of 500 attempts — found by scanning the trajectories for a rejected permission as the last tool call. The framework recorded those attempts ascompleted, because the harness exited cleanly; the export's terminated-reason column cannot see a harness that gives up on its own. - Claude Code: calls the Anthropic-format
/messagesendpoint, which is pre-flight token-counted by the router about a third higher than the OpenAI-format endpoints; a large context window declared to the harness will be rejected earlier than the model's real window. - mini-swe-agent: the reference agent needs to be told where the repository is; a one-line addition to its task framing, byte-identical to the CLI's own, was the difference between a working run and an agent searching the wrong directory.
- custom minimal: a deliberately small tool-calling loop used as the control for "how much does the scaffold add" — its own page: The custom minimal agent.
- all five: compact at the same context trigger, configured or implemented per harness — Context compaction.
Adapters live under swebench_eval/harnesses/.