Lessons

Everything here was paid for, in inference dollars, in wall-clock, or in a result that had to be thrown away. Each lesson links to the page that has the evidence.

Isolation and integrity

Pin what you can; build only what you must. The first image pipeline built environments and cloned repositories itself. The clone fetched the upstream tip before checking out the base commit, so one git log --all reached the gold patch, and the environments drifted between validation and grading: ten of twenty-two unresolved verdicts in one run were the environment, not the agent. Replaced by SWE-bench's official per-instance images, pinned by digest, each graded against the gold patch before a model dollar is spent. Images, Integrity.

Assume the network is on until the probe says otherwise. A Claude Code attempt downloaded the gold patch and the gold test file and was graded resolved. The verdict was discarded; the task subnets lost their default route, the only reachable endpoints are in the VPC, and an isolation probe now asserts that the clone is blocked before a run. Later, every stored trajectory was scanned for the mirror's hostname, /proc/1/environ and the credentials endpoint: zero. Integrity.

A flag without a cited turn is an opinion. The first judge flagged five contamination cases on one run; all five were ordinary reads of history at or before the checkout. The rubric now demotes any score above baseline that has no cited evidence, and the judge has to name the turn. Under that rule, twelve flags in 2,509 attempts, four at top severity, all of the memorisation class. The judge, judge results.

A detector's map is a claim too. The offline leak detector works from a map of hidden test ids "absent at the base commit", built by parsing the gold test patch. A def line the patch modifies looks like one it adds, so fourteen instances hit on all five harnesses because the agent opened the file. Classifying where a name first appeared, in the task, in a tool result, or in the agent's own text, is what turned 123 hits into 39 worth reading and one worth flagging. Integrity.

Never render a missing number as zero. Every numeric field on this site is nullable and a null says not measured. A detector that never ran must not read "0 leaks"; a pass that lost verdicts to timeouts must not look cleaner for it. The judge chart divides by attempts with a verdict, and the coverage is printed next to every number. Methodology.

Measurement

Meter from outside the thing you are measuring. Four of the five harnesses cannot enforce a spend cap. A local proxy in every task counts tokens, holds calls for the pacer and ends the session at the budget, for all of them the same way; the harness's own counters are a cross-check, never the authority. Without it, cost and turn caps would mean different things per harness. Admission control, Harness adapters.

Cheaper per token is not cheaper per task. Two similar models, the same harness, prompt, tools and limits: the one 5 to 10 times cheaper per token cost 4 times more per resolved instance, because it made 2.6 times the calls and carried 2.9 times the context on each. Only a per-call ledger shows it. The same lesson held for the tools that built the rig: DeepSeek was cheaper per token, Fable was cheaper per delivered feature. Model vs model, How it was built.

One line of harness configuration can be the whole result. mini-swe-agent searched the wrong directory until one sentence named /testbed; OpenCode's grep and glob tools had never worked in the isolated images and its permission system ended 23 of 500 sessions on its own; Claude Code's endpoint is pre-flight token-counted a third higher than the others. None of that is the model. Harness configuration.

A harness that exits cleanly hides its own failure. OpenCode's auto-rejected permissions were recorded as completed because the process exited zero; the export's terminated-reason column cannot see a harness that gives up on its own. It took a scan of the trajectories for a rejected permission as the last tool call to count them. Harness adapters.

Compare at the same trigger. Three CLIs compact their context natively, at three different points, in units that are not the same thing (one subtracts its output reserve from the window, one takes a token count, one needs a declared model limit or never compacts at all). Deciding the threshold once and rendering it five ways is what makes a cross-harness bill comparable, and the first attempt at one of them compacted nearly 40 K tokens early before a live run caught it. Context compaction.

Resolve rate cannot separate harnesses at n = 500. Five scaffolds on one model landed within two points of each other with overlapping intervals. Cost, calls, context and the judge's behavioural dimensions are where they differ, and those need both denominators and the ledger. Headline results.

Control loops and infrastructure

Ground truth is the cluster, not your memory. A planner that could not see booting tasks held every ramp back; a 60-second scale-in rule churned idle workers; managed scaling held a paid host at zero tasks. Every loop now reads ECS and the live keys each tick, staggers launches, and never kills a running task to scale in. Admission control, Eval scaling.

The provider's limit is unpublished and moves. It shed load at about 50 requests a minute for one key, changed edge about twice a minute, and never sent retry-after on a real overload, which is how the gateway's own 429s are told apart from the provider's. A discovery probe per pool, additive increase and multiplicative decrease, and a bias toward backing off are the answer; a fixed concurrency is not. Admission control.

Pin the provider, or the window is not a constant. Through a router, the same model name can resolve to providers with different context windows, throughput and token counting. The judge went from 40 to 50% timeouts at 23 tokens a second to none at 110 once the provider was chosen deliberately. Judge results, Cost.

Destroy the expensive tier every night. The idle stack costs about a dollar a day; a forgotten eval tier costs more than that per hour. Teardown is automated and verified, and the one time it failed was a login token expiring between two tiers; the fix, exporting credentials before the first destroy starts, is queued for the teardown script. Cost, Set it up yourself.

Your secret scrubber will refuse a real export. A hostname denylist matched a Django class attribute; a 12-digit account-id pattern matched the last group of a UUID in a test fixture. Each refused an entire run's export. Anchor the patterns, test the false positives, and print the context of a hit so the next one costs one look, not a session. Observability.

Process

Write the design down and make it the boss. A signed-off requirements and architecture record, decisions as ADRs, a task list with a definition of done per phase, and an independent reviewer session that verified each phase by running the checks rather than reading the code. That is what kept three or four builder sessions on one repository from drifting apart in four weeks. How it was built, Decision record.

Start on the cheap model. The first two days on the most expensive coding model burned a fifth of the inference budget for a fifth of the tokens the month's cheaper model delivered; the expensive model's turn came when the design was fixed and the work was review and integration. How it was built.

Publish your own mistakes first. A rig that measures other people's agents earns the right to by listing every flag it raised on itself, what each turned out to be, and which are still open. The Integrity page is that list. Integrity.