Skip to main content

Inference gateway (LiteLLM)

What it is. Every inference call from every agent passes through the gateway tier inside the VPC: LiteLLM, run as two replicas behind an internal load balancer at one stable name. It is where budgets become enforceable, where cost is metered from the provider's own usage figures, and where the pacer's ledger lives.

Replicas and shared state​

The gateway is not a single process. The replica count is a Terraform variable, two in the environment the published series ran on, and each replica is stateless:

  • Postgres holds the virtual keys, the alias registrations and the spend log, so a key minted through one replica is honoured by the other, and a call's cost lands in the same table whichever replica served it.
  • Valkey holds the pacer ledger and LiteLLM's own rate counters. Admission is a single Lua script that executes atomically inside Valkey using the server's clock, so the shims in 500 tasks are paced against one bucket per pool no matter which replica their call reaches. If Valkey is unreachable a shim falls back to slow local pacing and logs it loudly; it never runs unpaced.
  • The load balancer spreads calls across the replicas and drops one that fails its health check. A replica can be lost or added mid-run without a call's accounting or its admission changing.

Scaling out is the variable; the shared ledger is what makes the copies interchangeable.

Keys and aliases (ADR-0035, ADR-0044)​

  • One virtual key per run on the gateway, and one provider key per run minted at OpenRouter with the run's spend cap as its hard limit. ADR-0035 originally put the cap on the gateway key's max_budget; ADR-0044 moved it to the provider key, because the provider's own accounting is what actually bills, and the gateway key now carries no budget. Both keys are revoked when the run is closed. A judge pass, a discovery probe and a calibration run each get their own pair the same way.
  • One alias per (model, harness) pool, e.g. minimax-m2.5-opencode. The alias is what the pacer budgets, what the planner projects, and what the results carry as model_alias; the provider's actual model id is recorded separately as model_resolved.
  • Aliases that must rotate live are registered as database models, never in the static config (the two paths do not merge the way the docs suggest — found live).

Metering​

Every call lands in llm_calls with started-at, latency, time-to-first-token, paced wait, overload retries, status, tokens (prompt, completion, cached, reasoning), cost and the provider that served it. The site's per-instance call timelines come straight from this table. Prompt storage stays on for every run (owner decision) so a trajectory can always be re-read.

The pacer (ADR-0041)​

Per alias, in Valkey, a Lua script admits or holds each call against five constants: r_tok (arrival tokens/s), c_burst (arrival bucket capacity), k_inflight (in-flight token cap), r_qps (calls/s) and c_req (request bucket). A discovery probe seeds them by ramping a pool until it strains, then measures the cache-hit axis so cached prompt tokens are charged at their real cost. The operator dashboard shows the bucket fills, the wait queue and the provider's own 429s side by side, so "our pacer is holding" and "the pool is pushing back" are never confused.

Timeouts​

The gateway's load balancer holds a non-streaming call for at most ten minutes; a judge call that outlasts it is recorded as a timed-out judgement rather than retried. The harness path streams.

Gateway config: infra/docker/litellm_config.yaml; pacer: swebench_eval/gateway/pacer.py.