Requirements

These were written and signed off in August 2026, before the first line of code, and every build phase was reviewed against them. They are restated here without the internal cross-references. Each requirement carries the acceptance criterion it was held to and names the page that shows it met. Where the rig does not meet a requirement, or meets it only in part, the row says so.

The numbering runs F1 to F24 with F9 missing: it covered a second benchmark and was dropped when the scope was locked to SWE-bench, and the gap was kept rather than renumbering everything. F23 and F24 were added after the first version.

Goal

A public, open-source evaluation framework for SWE-bench that lets a user pick any agent harness and any model independently, run the grading at scale on AWS, and reproduce results others can trust.

Functional requirements

F1 — Pluggable agent harnesses. Any of a custom-built minimal agent, OpenCode, Claude Code, Codex CLI, Aider and mini-swe-agent must run as the component that turns a repository checkout plus an issue into a candidate patch, behind one common interface. Acceptance: adding a harness means one adapter (a Docker image and a run function) and no change to the orchestrator, gateway, evaluation or reporting code; mini-swe-agent was the intended proof. Today: six adapters exist; five ran the published 500-instance series, and mini-swe-agent was added as a single adapter. Harness adapters.

F2 — Model choice decoupled from harness choice. The model and provider are selected independently of the harness, through a shared inference gateway. Acceptance: swapping the model is a configuration change with zero code diff in any adapter, including for harnesses with strong vendor ties, so Claude Code must run against a non-Anthropic model unchanged. Today: every harness reaches its model through one gateway alias; the series ran Claude Code and Codex CLI on MiniMax. Inference gateway, Harness configuration.

F3 — Configurable rate and throughput limits. The gateway enforces per-model limits on requests, tokens and in-flight calls, tunable without code changes. Acceptance: two run configurations pointing the same harness at the same model with different ceilings observably produce different throughput and queueing. Today: a token-bucket pacer per model pool, seeded by a discovery probe and editable live from the dashboard. Harness admission control.

F4 — Centralised cost and usage tracking. Tokens and cost are attributable per run, harness, model, instance and attempt, sourced from the gateway rather than from each harness's own reporting. Acceptance: the report breaks cost down per harness and model and drills into a single instance without any harness having implemented its own instrumentation. Today: a shim in every task meters every call into a per-call ledger; adapter-reported figures are stored only as a cross-check. Observability, Cost.

F5 — Grading comparable to published SWE-bench results. Patch grading reuses the official SWE-bench evaluation harness unmodified, never a reimplementation. Acceptance: the verdict for an instance and patch matches what the official harness produces directly. Today: the official 5.0.2 harness runs inside the official per-instance images, pulled by digest, and any published run can be regraded with the official harness from the downloads. Evaluation and grading, Reproducibility.

F6 — Scale to SWE-bench Lite on AWS. The full Lite split, 300 instances, across at least one harness and one model, with sandboxed grading on Terraform-provisioned AWS infrastructure, within a $100 to $300 compute budget for the pass. Today: exceeded. The rig ran the full Verified split, 500 instances, through five harnesses; the measured compute and inference figures are on the cost page. Cost.

F7 — Two independent, observable capacity limits. A compute ceiling (how many containers run at once) and an inference ceiling (how fast calls reach a model), each constrainable and observable on its own, each enforced by an automated mechanism rather than sized by hand per run. Acceptance: a run can show queueing from the compute ceiling with inference unconstrained, and separately from the inference ceiling with compute unconstrained. Today: the harness fleet has a planner and a pacer, the eval fleet has its own scaler, and each pool has its own pause and its own charts. Harness admission control, Eval scaling.

F8 — Results and artifacts persisted and reportable. Per instance and attempt: the patch, the full harness transcript, the verdict, wall-clock time and reconciled usage; a reporting step aggregates them per harness and model. Today: artifacts per attempt in object storage, 56 columns per attempt in the database, and an export that reports both denominators. Observability, Downloads.

F10 — Publishable. The repository, documentation and diagrams are clean enough to publish on GitHub and to explain in an article for engineers who want to run their own evaluations. Today: this site, and the repository being published from the validated tree. Code.

F11 — Web UI for the run lifecycle. Launch a run (split, harnesses, model alias, limits, attempts per instance), watch per-instance progress live, and browse past runs. Acceptance: start a run, watch instances move through the pipeline and open any instance's detail without a CLI or a database query. Today: the operator dashboard; the Explorer on this site replays it on real runs. Controls and observability.

F12 — Instance-level drill-down. From the UI, open an instance and see the patch as a diff, the raw harness log, the turn-by-turn trajectory and the evaluation report, and pick the attempt when there is more than one. Today: the instance page shows all four, plus per-phase timing, the per-call table and the test output. Controls and observability.

F13 — Trajectory preservation for a later LLM judge. Every attempt persists a normalised, harness-agnostic trajectory (messages, tool calls, results, timestamps) alongside the harness's native log, so a judge can assess legitimacy across harnesses in one format. Acceptance: a Claude Code trajectory and an OpenCode trajectory for the same instance share a schema despite very different native logs. Today: every adapter writes the same trajectory file, and the judge reads it. The judge, Integrity.

F14 — Error taxonomy. Distinct outcome categories per attempt (resolved, unresolved, harness timeout, harness stuck, harness crash, gateway error, empty patch, patch failed to apply, evaluation infrastructure error, orchestration error) rather than a single pass or fail bit. Acceptance: the report and UI filter and count by category, and a legitimate unresolved is distinct from an infrastructure failure that should be retried. Today: Failure taxonomy.

F15 — Respect SWE-bench timing constraints. The official harness's test-execution timeout is passed through unmodified, never shortened. The agent side has its own separately configurable timeout, and wall-clock time is recorded for both phases. Today: grading runs under the harness's own 30-minute test timeout; the harness timeout is a run setting; both phases are timed on the instance page. Limits actually enforced.

F16 — Restartable runs. An interrupted run resumes from persisted state, re-running only attempts not yet in a terminal successful state, without duplicate cost or rows. Today: run state lives in the database and the queues, not in a process; graded instances are never re-run, and the operator restarts the remaining ones from the dashboard as a further attempt. Controls and observability.

F17 — Efficient reuse of cacheable resources. Repository checkouts, environments and built images are cached and reused across instances and runs, with explicit invalidation tied to the pinned harness version. Today: one image per instance, built once into a private registry and pinned by a committed digest snapshot; a framework change rebuilds only the last layer. Images.

F18 — Repeated attempts per instance. An explicit attempts per instance run setting, tracking each attempt as an independently graded unit from day one. Acceptance: three attempts produce three stored trajectories, patches and verdicts, and the report shows both a single-attempt number and a pass@k number without re-running. Today: a run setting; the export carries pass@k. Observability.

F19 — Cost estimation, upfront and post-hoc. An estimate before launch with a stated confidence tier, and actual cost after, reconciling an immediate figure against billing data. Today: not met for a run launch. There is no cost estimate before a new run is committed; the run's budget cap is the control, and the cost is known exactly from the ledger as the run proceeds. Cost previews do exist before the two optional spends, the judge pass and the discovery probe. The compute-cost reconciliation is not built either: the columns exist and nothing writes them. Cost, Capabilities.

F20 — Detect and bound non-productive execution. Beyond a wall-clock ceiling, detect an agent that repeats actions without progress, and distinguish that from one still working when time ran out. The requirement staged this: detect-and-log first, promoted to an active kill only after validation against real trajectories. Today: the first stage. The detector runs on every attempt and logs its verdict. The kill was deliberately not promoted: the per-instance turn cap and dollar cap already bound a looping agent tightly, and they are exact where a loop heuristic is a guess, so the action part would have added a false-positive risk for no extra protection. Limits actually enforced.

F21 — Per-instance live spend enforcement. An attempt's cumulative tokens and cost are tracked turn by turn while it runs and the attempt is terminated when it crosses a configurable ceiling, for every harness, with no detect-only mode. A missing ceiling fails closed. Today: the shim in every task enforces turn, token, dollar and context-window caps and refuses the call that would breach; the live total is visible while the attempt runs. Limits actually enforced.

F22 — Operator run control. Pause is reversible and system-wide: nothing new is picked up, in-flight work finishes, idle compute scales down. Abort is per run and terminal: in-flight tasks stop, queued work is discarded, partial patches are kept, and the run does not resume. Neither depends on the UI being reachable. A stopped run reports completed, aborted-in-flight and never-dispatched counts separately and never quotes a resolve rate over instances it never attempted. Today: pause and resume per pool, abort with real sweep semantics, a durable control state that reads as paused when stale. Controls and observability, Denominators.

The two requirements below were added after the first version shipped, once real trajectories existed to design against. The first version had deliberately left both out of scope.

F23 — An LLM judge over the trajectories. A rubric-driven judge that reads the normalised trajectory of every attempt and assesses how the agent worked, not only whether the tests passed: whether it solved the task legitimately, how it explored, where it wasted calls. Every finding must cite evidence a reader can open in the trajectory; a score without evidence does not count. The judge runs offline from stored artifacts, so its findings can be regenerated by anyone with the downloads, and its cost is previewed and confirmed before it runs. Acceptance: a judged run has a per-attempt verdict per rubric dimension with cited turns, a pass ledger that records what was judged and what failed, and a synthesis a human can check against the citations; a second pass on the same artifacts reproduces the findings. Today: nine rubric dimensions, evidence required and scores without it demoted, a resilience cascade for empty judgments, a pass ledger with a lock, calibration reviews with mandatory reasoning, and a leak pass that runs first. The judge, Judge findings.

F24 — Contamination and leak detection. The rig must make it hard for an agent to see the answer and must detect it when an agent games the grade anyway. Prevention: the agent's container cannot reach the gold patch, the future of the repository's history, or the network. Detection: every attempt is checked for edits to test files, for similarity to the gold patch, and for the answer appearing in the trajectory; the eval side strips test-file changes before grading. A flagged attempt is marked, never silently counted, and the published numbers state both denominators. Acceptance: an attempt that edits a test or reproduces the gold patch is flagged in the results and in the export; the checks run offline from the stored artifacts; the site publishes every flag it found, including the ones that turned out to be false positives. Today: Integrity, Denominators.

Non-functional requirements

N1 — Secure by default. Private subnets for compute, least-privilege IAM per component, encryption at rest, secrets in a secrets manager and never in Terraform variables or task definitions, security groups scoped to what each component needs. Today: the harness subnets have no default route at all, tasks carry no secrets, both load balancers are internal, and the only operator path is a tunnel. Integrity.

N2 — Cost legibility. Every infrastructure choice with a real cost implication is documented with its cost trade-off, not only its technical one. Today: Cost, and the setup guide's per-step time and cost table. Reproducibility.

N3 — Config-driven experimentation. Harness, model, limits, split, concurrency and attempts are run-configuration fields; a new experiment does not touch the code. Today: all of them are launch-screen fields. Capabilities.

N4 — Observability. Structured logs from every component to one place, and both capacity axes visible as live metrics rather than reconstructed afterwards. Today: Observability.

N5 — Resilience. Transient failures retried with backoff, reusing library and SDK support; every long-running operation under a forceful timeout, including an aggregate per-run cost ceiling, so no failure mode becomes unbounded spend. Today: the shim retries provider overloads so the harness never sees one, every phase has a timeout, and the per-run budget is the provider key's own hard limit. Limits actually enforced.

N6 — Scale-ready without building scale early. No component has a ceiling that only a rewrite can raise. The shape (address ranges, where state lives, statelessness) is built for the target from the first apply; the size (replicas, capacity units, node classes) stays at the minimum until a run needs more. The concrete target: the full split at up to 20,000 concurrent instances, reachable by changing variables rather than design. Today: designed for that shape, operated at 500 concurrent, stated with that qualification. Architecture.

N7 — Build-phase cost containment. Development and testing are cost-contained independently of run-level budgets: an account-scoped spend alarm, an environment sized and torn down independently, and lifecycle policies on storage. Today: a budget and a billing alarm, tiers torn down nightly with a guard that refuses unless everything is idle, and lifecycle rules on the registry. Cost.

Explicit non-goals

• Reimplementing or modifying SWE-bench's own grading.
• Every split on day one. Lite first, with Verified and Full as a configuration and capacity change rather than a redesign. The rig went on to run Verified in full.
• Multi-region or high-availability deployment. One region is acceptable for a batch workload.
• A hosted service. This is a framework others run in their own account.
• Building the LLM judge in the first version. The first version captured the data a judge would need; the judge was designed and built once real trajectories existed, and is now F23.
• Authentication for the web UI. The UI is never exposed publicly: its load balancer is internal and it is reached only through a tunnel, so it is single-operator with no public surface. If it ever becomes reachable without that tunnel, authentication becomes mandatory first.
• Any benchmark other than SWE-bench.