Images and environments

The problem. An agent's environment decides half of what "unresolved" means. If the testbed is missing a binary, the failing test is the framework's fault; if the repository's own history contains the fix, a passing test is meaningless. The framework went through one wrong answer on this before the right one, and the reversal is recorded rather than hidden.

The reversal: from building environments to pinning official images

The first pipeline built one environment image per repository version and prepared the checkout at task start with the framework's own clone. It leaked and it drifted. The clone fetched the upstream tip — which contains the merged fix — before checking out the base commit, so one git log --all retrieved the gold patch. And the environment specs were never a lockfile: in one run, ten of twenty-two unresolved verdicts were the environment (a pandas built against a newer numpy, missing django admin templates, a floated pytest that killed 192 pass-to-pass tests).

The decision (ADR-0043, 2026-09-05): use SWE-bench's own published per-instance images, one per instance, pulled by digest from a committed snapshot, because the upstream :latest tag is mutable. The official image freezes the non-Python libraries too, and carries SWE-bench's own hardening of the repository: origin removed, tags after the base deleted, reflog expired, garbage collected, and an assertion that zero future commits remain. Adapters are forbidden from touching the checkout. (MiniMax ran git log --all and git show unprompted in the published series; it found nothing, because the hardening held.)

Rejected on the way: pinning pandas<3; deriving pip constraints from the specs; staying on SWE-bench 4.1; trusting :latest; hand-correcting the dataset.

The layer tree

Each layer has one job. The CLI layer pins the harness CLIs (mini-swe-agent 2.4.6, Claude Code 2.1.234, Codex 0.147.0, OpenCode 1.18.18) and creates the unprivileged agent user the harness runs as. The base layer writes a sentinel recording the instance id, base commit, framework commit and the digest it was built from — the worker refuses to start if the sentinel's instance id is not the job's ("A's repository against B's tests" is a hard failure, not a warning) — and bakes a wheelhouse with PIP_NO_INDEX=1 so that grading can never reach PyPI. The framework layer is thin and always last: a framework change rebuilds only that layer.

Pins, and how a report states them

Three things pin a run and every report names all three: the dataset revision (SWE-bench/SWE-bench_Verified @ 78f471bf655a), the image-digest snapshot file (500 entries; the snapshot tool refuses to overwrite and has a --check mode that reports upstream tags that moved), and the harness version (swebench 5.0.2). Every -inst image records the CLI versions from the running binaries and the digest of the official image under it. Reports never mix triples.

Building and refreshing 500 images

• Build tier: a separate ECS cluster (editing the eval cluster's capacity providers would re-plan the eval fleet) of Spot c5d.4xlarge hosts on public subnets with public IPs, because a terabyte of Docker Hub pulls through NAT costs about $45 or 35 hours through the small NAT instance. Work is sharded by repository environment with longest-processing-time packing weighted by measured seconds per repository (matplotlib 742 s down to a default of 306 s), and resumes from what ECR already holds. A full official build: 459 images in 97 minutes, no failures.
• The refresh short path: pull the existing -inst from ECR, apply only the framework layer (a uv.lock comparison guard decides whether the full path is needed), carry provenance forward. 477 images in 64 minutes on four hosts. Promotion is an ECR manifest copy — no layer traffic — and the previous digest is kept as -inst-prev; nothing in the pipeline can delete an image.
• Admission: per-instance records are merged into a manifest that drops any tag ECR no longer serves at the recorded digest. The dispatcher launches an instance only if its manifest entry is present and the recorded base digest equals the snapshot's; a task family that is not registered is refused, never defaulted.

The gold gate

No -inst image is launchable until the gold patch grades RESOLVED in it — implemented as a run with status = 'validation' that reuses the regrade path with use_gold_patch = true. On the Verified set (2026-09-06, three gate runs): 499 passed on the first pass; pylint-dev__pylint-4661 failed because its gold patch adds a dependency the offline image could not install — fixed by an "extra wheels" mechanism that downloads, but never installs, a named wheel — and passed on the third; 500 of 500. Sixteen grades in the first 500 gate were abandoned by disk exhaustion on the eval hosts, which is where per-grade image release and eviction came from.

One caveat the gate cannot close: matplotlib__matplotlib-20488 validates only because the eval tier has network egress. It is flagged in the methodology rather than papered over.

Isolation findings

An audit of the harness container from the inside, after the published series, found three gaps and closed them:

gap	what an agent could reach	fix
G1	a full-history git mirror service, reachable from the harness security group — the gold patch one hop away	security-group rule and environment variable removed; the isolation probe now asserts the clone is blocked; the mirror service is deployed but vestigial
G2	as root, /proc/1/environ → task credentials → prior patches, the results queue, the gateway master key	non-root agent user, runtime chown of the testbed, no secrets in the task definition, the execution role's secret grant removed
G3	dependency re-solve at build time drifting the environment	ADR-0043 official images plus the gold gate

One remains open and is stated: G4, test gaming through configuration files that the touches_test_files check does not cover.

Environment defects left as official behaviour

django__django-10097 is a lottery in the official image (an unsorted directory listing, a settings leak and an SQLite rename) and grades nothing the patch changes; it is flagged known-bad in the methodology rather than patched. The only environment exceptions are fenced and named: a downgrade-only pip guard for 32 scikit-learn instances, and the extra-wheels mechanism above.

Decisions

decision	why	record
official per-instance images, pinned by digest	the specs were never a lockfile; :latest moves	ADR-0043
cache them in a private registry, warmed by a job	Docker Hub rate-limits per source IP; 500 pulls per run	ADR-0020 (image source superseded by 0043)
one task family per instance	the image is the instance	ADR-0030
the harness never clones	the clone leaked the fix	image pipeline design, ADR-0033
a separate build cluster on public subnets	NAT cost, and the eval fleet's capacity providers stay untouched	build-tier notes

Code: scripts/warm_image_cache.py, infra/docker/Dockerfile.instance-layer, infra/docker/Dockerfile.harness-worker-refresh, swebench_eval/orchestrator/control_plane/image_validation.py.