Skip to main content

High-level architecture

What it is. A control plane on AWS that turns "this harness, this model, these instances, this budget" into 500 isolated agent tasks, grades every patch in the official SWE-bench environment, meters every inference call, and lands one durable row per attempt. It is designed for the full SWE-bench split at up to 20,000 concurrent instances without re-architecting (ADR-0018), and has been operated at 500.

The harness task reaches Valkey, S3 and SQS only through in-VPC endpoints; its route table has no default route. The repository is baked into the instance image (SWE-bench's own install_repo_script, history pruned at the base commit), so the task clones nothing at run time. The eval job is enqueued by the results-writer when the harness row lands as PATCH_READY — an empty patch never reaches the grader. The judge is a separate ECS task run after the fact; it reads the trajectories from S3, calls the model through the same gateway under its own key, and writes verdicts to Aurora. The image-build and warm-up pipeline (a separate EC2 build tier feeding the registry) is described under Evaluation & grading.

Inside one harness task: the local proxy shim​

Every agent task carries a second, tiny gateway of its own. The harness process is pointed at 127.0.0.1, where a local proxy shim (ADR-0019, ADR-0037) sits between the agent and the fleet gateway. Four of the five harnesses cannot enforce a spend cap themselves; the shim does it for all of them the same way, and it is where every call is metered before it leaves the container.

What the shim does on every call, in order: ask the pool's pacer in Valkey for admission (and wait if the bucket is empty); forward the request with the run's key; read the provider's usage from the response and add it to the attempt's running totals; write the live progress key the dashboard reads; and, if a cap is now exceeded, end the session so the attempt lands as budget_exceeded or max_turns_exceeded with whatever patch exists. Provider overloads are retried with backoff here, so the harness never sees a 429. Cached prompt tokens are charged to the pool at their real, discounted weight. The ledger it writes — one row per call with tokens, cost, latency, pacer wait, retries and status — is the source of every cost figure on this site.

The shape, in five decisions​

  1. Harness and eval are different fleets (ADR-0012). Agent tasks are Fargate containers with no egress; grading needs Docker-in-Docker on a real host, so eval workers are EC2. The forcing line is one Terraform attribute: infra/terraform/modules/ecs-service-eval-worker/main.tf:215 — requires_compatibilities = ["EC2"]. See Evaluation & grading.
  2. Every inference call goes through the gateway tier, and the tier is replicated (ADR-0002, ADR-0035, ADR-0044). Two LiteLLM replicas in the VPC behind an internal load balancer, both stateless: keys, aliases and the spend log live in Postgres, the pacer ledger and the rate counters in Valkey, so either replica can serve any call and one can be lost or added without a call's accounting changing. One virtual key per run, and the run's spend cap enforced as the limit on a per-run provider key (ADR-0044 moved the cap there; the gateway key carries no budget of its own). One alias per (model, harness) pool. The gateway's own metering is the cost figure, not the harness's guess.
  3. Admission is paced, not hoped for (ADR-0041, ADR-0019). A token-bucket pacer in Valkey admits each call against the pool's discovered rate limits; a planner in the dispatcher sizes the fleet from the pacer's ledger. Four of five harnesses cannot enforce a budget themselves, so a local proxy shim in each task does it for them.
  4. Live state lives in Valkey, durable state in Aurora (ADR-0018). Progress keys with a TTL for the dashboard; one row per attempt, phase and call for the record. The dashboard never queries Aurora for liveness.
  5. Environments are pinned and validated (ADR-0020, ADR-0043). The official per-instance images (~1.1 GB each) are mirrored to a private registry by digest, and every image is gold-patch validated before a run. One task per instance is ADR-0030; the dispatcher hands the task a job reference rather than the job (ADR-0032).

Scale facts that shaped it​

  • 500 concurrent Fargate tasks pull nothing from Docker Hub; the harness image is one layer on top of a cached base, and the per-instance eval images are pulled from the private registry.
  • The VPC is sized for 20k ENIs; the queues, the pacer and the progress keys are per-run and per-alias so two runs can share a pool without sharing a budget.
  • Idle cost is dominated by the registry (about 2 TB of instance and harness images across the Lite and Verified sets — $20 of the $317 AWS bill for the project's first month); the compute tiers scale to zero and are torn down nightly by a guard. See Cost.
Claim discipline

"Designed for 20,000" is a designed shape; 500 is the demonstrated one. The ENI quota was confirmed non-binding at the full split for five harnesses (~550 of 5,000) but a 20k run has not been executed (design appendix item 9).

The component pages that follow summarise the internal design record as the system was built. ADR numbers cited on these pages refer to that record, which is not published; the decision record index gives each decision's substance.