Controls & Observability
The operator dashboard is a React application served against the control plane's API over a private tunnel. The public Explorer is that same dashboard, unchanged in layout and controls, running in a demo mode that replays eleven real, exported runs from static files, with every control simulated in the browser: evalsui.preetdhaliwal.dev.
A bar under the header says which run is being replayed, at what speed and at what point in the run's clock, and that actions are simulated. Nothing you do there leaves the browser.
What to look at
| screen | what it shows | open |
|---|---|---|
| Runs | the run list with activity-gated polling; every published run is there with its provenance | runs |
| A 78-instance run replaying | OpenCode × MiniMax, 28 minutes of wall-clock: the live panel filling, the planner's binding constraint, the pacer buckets, the eval scaler following patch arrivals | 883d2571 |
| The admission-control loop under strain | mini-swe-agent × Laguna: five hours pacer-bound under the provider's request cap; overloads, cooldown and recovery visible in the pacer and autoscaler panels | c40be1f7 |
| A 500-instance run | OpenCode × MiniMax on the full split: 95 concurrent tasks at peak, both denominators, the queues | 020169d5 |
| Launch | the full launch form — dataset, instances, harness, model, limits, presets — resolving to the matching captured run | launch |
| Capacity and model ceilings | both pools' capacity history and the discovered ceilings per model | capacity |
Two buttons on the demo bar tell the honesty story: simulate Valkey outage turns the live panel to unknown and the control state to stale, fail-closed — every action disabled until the store is confirmed; simulate provider 429 burst makes the pacer's overload counter climb, the planner cut the token budget and enter cooldown, then grow back by 5% per clean window.
What the operator sees, panel by panel
- Run header: provenance, per-instance bucket counts, both denominators with the live resolve rate, cost against budget, tokens in and out.
- Live progress: every in-flight instance with its turn, tokens, cost so far, pacer state and age — Valkey-backed with a TTL, a supplement to the durable table, never a replacement.
- Pacer, the live admission ledger: per alias, the five constants, the token and request bucket fills, in-flight tokens, the wait queue with its head, admits and waits over 2 s in the last minute, and the provider's own 429s in a separate figure, so our back-pressure is never confused with the pool pushing back.
- Planner: mode, desired ceiling against in-flight, the binding constraint, booting tasks, hold-cap timeouts.
- Eval scaler: wanted against running grading tasks and hosts, queue depth, feed-forward, scale-in damping.
- Operator control: pause and resume per pool with an audit reason, abort with scope, gateway pause, close; per-run limit edits and per-alias pacer edits, each recorded as an event.
- Instances: search and filters by repository, phase, state, error category, judged or not, and judge finding; multi-select restart and regrade with explicit confirmation; per-instance drill-down with the diff, the trajectory, the call timeline and the eval report.
- LLM judge: launch with a cost estimate and confirmation, live progress, coverage including timed-out judgements, the rendered pass report, and the retry and regenerate controls.
What the demo cannot show
A few panels answer "not captured" rather than inventing a value: the work-queue depths and dead-letter counts (not in the exported timeline), the discovery cost estimate, and the judge panels for three of the five 500-instance runs whose judge bundles were not captured at snapshot time. Per-instance call timelines and artifacts are replayed for four runs; the others show the ledger rollups only. The live LLM-call view is reconstructed from the ledger and the stored trajectory and says so on screen.