The custom minimal agent

The framework ships its own harness, and it is deliberately the smallest thing that can solve a SWE-bench instance: one loop, two tools, a four-step prompt. It exists as the control for the question every other harness raises — how much does the scaffold add? — and as the reference implementation of the harness contract, the first adapter built and the one that proved the interface and the grading reuse end to end before anything else depended on them.

On the five-harness comparison it resolved 384 of 500 at $33.98, inside the noise of the four full CLIs. Headline results.

How it compares with the four CLIs

Same model (MiniMax M2.5), same 500 instances, same caps. Each row names the best of the five on that measure, so the control can be read against the field and against the leanest CLI.

per attempt unless stated	custom minimal	OpenCode (cheapest CLI)	best of the five
resolved of 500	384	385	385, OpenCode; all five within the interval
cost per resolved instance	$0.088	$0.068	$0.068, OpenCode
cost per attempt	$0.068	$0.053	$0.053, OpenCode
agent time, median	161 s	142 s	126 s, Claude Code
calls, median	43	29	29, OpenCode
fixed context on every call	2.6K tokens	7.5K	2.4K, mini-swe-agent
prompt tokens per call	25.4K	25.0K	23.1K, mini-swe-agent
output tokens	11.5K	8.7K	8.7K, OpenCode
test runs, mean	8.8	5.7	5.5, Codex
test re-runs with nothing changed, mean	0.74	0.27	0.24, Codex

Head to head with OpenCode, the two resolve 361 of the same instances; 23 only the control, 24 only OpenCode. Five instances were resolved by the control and by none of the four CLIs (django-11265, django-11477, django-11885, pytest-7205, sphinx-10435); nine were resolved by every CLI and not by the control. It covers 90% of the union of everything any harness solved. Head-to-head.

What the judge saw, reading every attempt with cited turns (LLM judge):

attempts flagged, of about 500	custom minimal	OpenCode	best of the five	worst of the five
hallucinated tool output	25	63	25, custom	104, Claude Code
gave up early	12	39	12, custom	62, Claude Code
environment problem	119	221	119, custom	221, OpenCode
looping	77	72	59, Codex	191, Claude Code
redundant tool use	139	82	52, Codex	258, Claude Code
redundant test runs (token-efficiency cause)	387	351	287, Claude Code	387, custom
test-file edits (test gaming)	21	3	0, Codex	21, custom

Read together: the control is the most honest agent in the set and the least likely to quit, it has nothing in its environment to break, and it solves as many problems as the CLIs. It pays for that with the most calls in the set, the most test runs and re-runs, the second-most redundant tool use, and the most test-file edits, which the evaluation strips before grading but which cost turns and draw integrity flags. Its fixed context is a third of OpenCode's, so its extra cost is entirely the extra calls: 43 against 29, and 11.5K output tokens against 8.7K.

The loop

Each turn is one chat-completion call in the OpenAI format, sent to the local proxy shim on loopback, which meters it, applies the pacer's admission decision and forwards it to the gateway. The agent keeps a plain message list — system, user, assistant with tool_calls, tool results — and nothing else: no planner, no sub-agents, no memory, no summary of what it has done so far.

Two tools. bash runs a command in the repository with a 60-second timeout and output truncated to 8,000 characters. str_replace_editor views, creates and edits files with exact string replacement, line insertion and undo; paths are relative to the repository root. Search is whatever the model types into the shell; there is no grep or glob tool to get wrong, which is one reason the control was immune to the ripgrep defect the judge found in OpenCode.

One prompt. The system prompt tells the model where it is (the repository is the working directory; do not cd to /repo), names the two tools, gives four steps — explore, find the root cause, make the minimal fix, verify with the relevant tests — and asks for the word DONE on its own line when finished. That is the whole prompt. The prompt-efficiency experiment appended five working rules to OpenCode's prompt, not to this one; the control stays fixed.

What the framework builds around it

The loop is small; the machinery that makes it a fair participant is not.

concern	what the harness does
limits	before every call it checks wall-clock, cumulative tokens and cumulative dollars against the run's per-instance limits, and stops with timeout or budget_exceeded; the shim enforces the same limits from outside, so the harness's own counters are a cross-check, never the authority
turns	a safety cap of 500 loop iterations (max_turns_exceeded); the run's real turn cap, 350 in the published runs, is the shim's
retries	the OpenAI SDK's own retries are switched off (retry policy lives at the gateway); the loop retries 408, 429, 5xx and network errors itself with jittered backoff from 1 s to 12 s, under a per-call read budget so a retry can never outlive the instance's wall-clock
broken tool calls	a provider that truncates a response mid-argument produces unparseable JSON; the harness hands the parse error back to the model as the tool result and lets it re-issue the call, and terminates with malformed_tool_calls after three in a row rather than looping to the turn cap
accounting	cost comes from the one shared pricing module the shim uses; every trajectory line carries the shim's live cumulative usage, so the harness's numbers and the ledger's cannot drift apart
trajectory	a normalised trajectory.jsonl is streamed live, one line per model call and tool result, with the model's reasoning content preserved; the judge and the efficiency profile read this format for every harness
context	a deterministic compaction pass runs at the top of the loop, before the call, when the measured context nears the window; Context compaction
exit	DONE ends the run; the patch is the diff of /testbed against the base commit, classified the same way as every other harness (empty, binary-only, or a patch)

Temperature is sent only when the run configures one; the harness never injects a default, so it samples exactly as the other harnesses do through the same alias.

What it deliberately does not have

No file-search tool, no planning step, no parallel sub-agents, no automatic test discovery, no prompt caching tricks, no summarisation of earlier turns beyond the compaction pass. Each of those is something a full CLI adds, and the point of the control is to measure what they buy. On MiniMax M2.5 at n = 500 the answer was: nothing the resolve rate can see, and a bill in the same band as the CLIs.

What would make it better

Ranked by the evidence above, largest expected effect first. None of these has been run; the control stays fixed so the comparison stays clean, and each would be a new arm.

1. Enforce "stop after green" in the loop. The control runs the most tests per attempt (8.8 against 5.5 to 6.8) and is flagged for redundant test runs on 387 of 501 attempts, the most in the set. The prompt experiment showed a stop rule is advisory when asked for in a prompt: the post-edit tail on OpenCode stayed at a third to a half of spend under the rules. This loop owns the turn, so it can do what a prompt cannot: notice a passing test run after an edit and end the attempt, or refuse further tool calls until the model says DONE. The control's 14 extra calls per attempt over OpenCode are the ceiling on what this can recover, and the tail is where most of them are.
2. Refuse edits to test files. Twenty-one test-gaming flags, seven times OpenCode's and the most of any harness. The prompt says "verify with the relevant tests" and nothing about not changing them. The evaluation strips test edits before grading, so the score is safe, but each one is a wasted turn and an integrity flag. One sentence in the prompt and a path guard in str_replace_editor would remove them.
3. Give it a bounded search and a windowed read. Search today is whatever the model types into bash, with output cut at 8,000 characters; the judge flags probing without searching on 313 attempts and redundant tool use on 139, second only to Claude Code. A grep/glob tool scoped to a directory and a view that requires a line range are exactly the reading rules that cut OpenCode's whole-file reads by half in the prompt experiment, here enforced by the tool surface rather than requested.
4. Run the working-rules preset that already exists. custom_minimal-efficiency-v2 is versioned in the framework beside OpenCode's and has never been run. By analogy with OpenCode on the same model, 20 to 30% off the bill at no change in resolve rate is the expectation, and it is a configuration change, not code.
5. Leave the rest alone. Its fixed context is already the smallest bar one, its hallucination and gave-up-early counts are the best in the set, and its time is mid-field. The gap to the cheapest CLI is calls, not context, and items 1 to 3 are all about calls.

Code: swebench_eval/harnesses/custom_minimal/harness.py (the loop), swebench_eval/harnesses/custom_minimal/tools.py (the two tools), swebench_eval/harnesses/custom_minimal/trajectory.py (the trajectory writer).