Observability: what is recorded, and what is only watched

The principle. The ledger is the record; the live view is advisory. Every number on this site comes from a row that was written once, by one writer, at the moment it happened. Live dashboards read faster stores with time-to-live keys, and when they cannot confirm something they say unknown, never zero.

What is recorded

record	one row per	what it carries
llm_calls	inference call, including refusals and failures	request shape; usage and cost with a cost_source of provider or local pricing; a latency decomposition where wall-clock ≡ preflight + pacer wait + retries + backoff + provider latency; pacer wait, deny axis and charge; overload retries; whether max_tokens was injected; whether usage parsing failed. No prompt or response bodies. Shipped from the task as one S3 pointer per attempt on its own queue, never on the results queue
instance_results	attempt × phase	the shim's rollup and the adapter's own counters side by side as a cross-check; turns used against call count; harness and eval timing (queue, provision, pull, boot, prepare, agent, extract, upload; grade queue, fetch, test, upload) with task-observed against task-billed seconds; the pacer rollup; leak fields with a leak_scan_at so "never scanned" is distinguishable from "clean"; compaction counters; artifact pointers; the state-rank guard
run_summary	run	recomputed from the rows, never incremented: both denominators, cost, tokens; empty patches stay gradeable
run_timeline_tick	run × 30 s	live and landed cost, per-alias pacer ledger, control flags including staleness, with zero extra Redis reads
capacity_snapshot	pool × 30 s	the planner's whole decision as JSON, its age, the source of its constants, utilisation, queue depths, ETAs as ranges
run_events, operator_limit_edits, control_state	event	every pause, resume, abort, limit edit and pacer edit, with actor and reason
model_tpm_observations	discovery observation	append-only; the ceilings view is derived
S3, per attempt	file	patch, normalised and native trajectory, stdout, the call ledger, compaction events; eval report, test output, run log; raw judge responses
gateway spend log	call	LiteLLM's own row with the stored prompt, kept on for every run so a trajectory can always be re-read; its spend column is never summed (it over-counted 3.5× against provider billing)

NULL means not reported. Nothing coerces a missing number to zero, on the row or on this site.

Live views are fail-closed

• /live returns state = unknown when Valkey is unreachable — never an empty list that would read as "nothing running".
• /control reads all paused when its snapshot is older than 90 s. The dashboard renders that as an amber "control state stale, fail-closed" banner and disables every action.
• /pacer and /autoscaler distinguish absent (no decision yet) from unknown (store down).
• Progress keys carry an updated_at; anything older than the staleness window is drawn grey.

The dashboard's components encode the same rules: three-state control rendering, "expected" and "gradeable" as two labelled numbers rather than one, ETAs as ranges, dead-letter counts above zero rendered as an alarm, per-panel error boundaries so one broken view cannot blank the page.

Exports, and what this site reads

Two endpoints feed everything here. /export produces the run JSON: the provenance block with the dataset and image pins, totals with Wilson intervals, pass@k, and one row per instance; infra retries are collapsed so spend is summed and outcomes taken whole. /timeline produces the raw ticks, capacity decisions, events and calls. The site exporter prunes both to an allow-list, scrubs against a deny-list of account ids, ARNs, subnets, hostnames and key ids — and fails on a hit rather than redacting — decimates series to 720 points, and writes a manifest with a hash per file. Re-runs are byte-identical. The unscrubbed raw bundle is kept separately and labelled.

How cost is resolved

Two tiers: the provider's reported cost wins; otherwise a pinned, hand-maintained price table with per-model cache-read ratios, where unknown models are priced high on purpose. The harness's own cost figure, where it has one, is stored beside it as a cross-check (Claude Code's was 30× wrong). Provider cost is also captured raw.

What is not measured, stated plainly: the three-tier cost estimator is a placeholder; the estimated-cost and confidence-tier fields, and the per-run compute cost columns, are read by the export and the UI and written by nothing. This site never shows compute cost as a measurement.

Alarms and the nightly guard

Ten log groups at 90 days' retention. Exactly two CloudWatch alarms exist: the billing alarm and NAT recovery. There is no alarm on queue depth, dead letters, task failures or latency — those are read-path rules in the dashboard, and an operator who is not looking is not alerted. The nightly scale-to-zero guard runs at 03:00 Pacific and refuses to act unless the database, the queues and ECS are all idle; it is disabled by hand for scored runs. The operator's tunnels are fixed SSM documents with no parameters, so the scoped policy cannot be pointed at the database.

Code: swebench_eval/orchestrator/control_plane/capacity_observer.py, swebench_eval/orchestrator/timeline_export.py, scripts/export_run_timeline.py, swebench_eval/orchestrator/api/llm_live.py.