Cheaper per token is not cheaper per task
Preetinder Dhaliwal · September 2026 · the numbers behind this piece are on Model vs model; the runs are replayable in the Explorer and downloadable as bundles.
Price sheets are quoted per million tokens. Bills are paid per task. Between the two sits the model's behaviour, how many calls it makes and how much context it carries on each, and no leaderboard number shows it. I measured it, with two models that look interchangeable on paper, and the cheaper one cost four times as much.
The setup
Laguna xs 2.1 and MiniMax M2.5: similar size, similar context window, both served through OpenRouter with the provider pinned. Per million tokens, Laguna is 5× cheaper on uncached input and 10× cheaper on output; cached input is priced the same on both, $0.03.
Both ran through the same gateway, under the same harness (OpenCode, then mini-swe-agent as a check), the same system prompt, the same tools, the same 78 SWE-bench Verified instances, the same limits: 350 turns, $2 per instance, a 262k context window, temperature 1.0. Cost is the gateway's metered per-call ledger at the provider's list price, not an estimate. Everything else is counted from the stored trajectories with one script for both models.
The result
| Laguna xs 2.1 | MiniMax M2.5 | Laguna is | |
|---|---|---|---|
| price per million tokens | |||
| uncached input | $0.06 | $0.30 | 5× cheaper |
| cached input | $0.03 | $0.03 | the same |
| output | $0.12 | $1.20 | 10× cheaper |
| the bill on OpenCode, 78 problems | |||
| resolved | 58 of 78 | 59 of 78 | the same, within the interval |
| inference cost | $15.45 | $3.49 | 4.4× more expensive |
| cost per resolved instance | $0.266 | $0.059 | 4.4× |
| the bill on mini-swe-agent, same 78 | |||
| resolved | 55 of 78 | 58 of 78 | the same, within the interval |
| inference cost | $15.55 | $4.37 | 3.6× more expensive |
| cost per resolved instance | $0.283 | $0.075 | 3.6× |
Cost per resolved instance, by model and harness
four runs · each model on each harness · same 78 instances, same limits, no prompt preset · lower is better
Inference spend only, from the gateway's own metering (compute cost is reported separately on the Cost page). Sorted cheapest first. A missing bar is an unmeasured run, not a free one.
Same problems solved, same problems failed: 55 of the 78 resolved by both models on OpenCode, 16 by neither. The resolve rate cannot tell these two models apart. The bill can.
Why: the bill is context × calls, and the discount is on neither
On OpenCode, Laguna made 2.6× the calls and carried 2.9× the context on each, so it sent 7.5× the prompt tokens for the same outcome:
| per 78 attempts, OpenCode | Laguna xs 2.1 | MiniMax M2.5 | ratio |
|---|---|---|---|
| LLM calls (median per attempt) | 6,514 (70) | 2,527 (23) | 2.6× |
| prompt tokens per call | 76.9K | 26.4K | 2.9× |
| prompt tokens in total | 501M | 67M | 7.5× |
| share of prompt tokens that are cache reads | 98.5% | 96.1% | |
| output tokens | 1.75M | 0.64M | 2.7× |
| median agent time per attempt | 447 s | 116 s | 3.9× |
Almost all of that context is cache reads: the conversation so far, re-sent on every call. Cache reads are the one token kind priced the same on both models. Splitting each bill by token kind at the list prices reproduces the metered total within one percent, and the split is the whole story:
| share of the bill | uncached input | cached input | output |
|---|---|---|---|
| Laguna · OpenCode ($15.45) | 3% | 96% | 1% |
| MiniMax · OpenCode ($3.49) | 22% | 55% | 22% |
| Laguna · mini-swe-agent ($15.55) | 2% | 96% | 2% |
| MiniMax · mini-swe-agent ($4.37) | 27% | 49% | 24% |
The 5× and 10× discounts apply to the four percent of Laguna's bill that is new text. The other ninety-six percent is a large conversation re-sent on 2.6× as many calls, at a price Laguna does not discount. A price sheet lists three numbers; an agent's bill is dominated by the one that is the same everywhere, multiplied by how the model behaves.
Where the calls go
Same tools, same prompt, same problems. The trajectories say what each model did with its turns:
| per 78 attempts, OpenCode | Laguna | MiniMax | ratio |
|---|---|---|---|
| whole-file reads (no offset or limit) | 503 | 97 | 5.2× |
| characters pulled in by those reads | 6.0M | 0.67M | 9.1× |
| re-reads of a file that had not changed | 728 | 245 | 3.0× |
| test runs / redundant re-runs of an unchanged test | 1,167 / 38 | 476 / 3 | 2.5× / 12.7× |
| call at which the first edit happened, mean | 24 | 9.6 | 2.5× |
| spend after the agent's own tests were green | $4.39 | $1.45 | 3.0× |
One model reads before it searches, re-reads what it already has, keeps testing after green, and takes two and a half times as long to reach its first edit. Every whole-file read lands in the context and is re-sent, as a cache read, on every later call; that is how a 5.2× difference in reads becomes a 7.5× difference in prompt tokens. An LLM judge over the same trajectories, required to cite the turn for every flag, ranked the causes independently and in the same order: repeated reads, redundant tests, probing before searching. Model behaviour.
The harness is not the explanation
Swapping the harness on the same model moved MiniMax's bill from $3.49 to $4.37 and Laguna's from $15.45 to $15.55: a quarter at most. Swapping the model moved the bill by a factor of four on either harness. The pattern is the model's; the second harness is the check that it is not an OpenCode artefact.
How much of it is instructable
Five working rules appended to the harness's system prompt, search before reading, open files with an offset and a limit, bound shell output, use the repository's runner on the covering tests, verify once and stop:
- Laguna: cost −36% (bootstrap 95% interval 17 to 51%), cheaper on 64 of 78 problems; whole-file reads 503 → 276; resolve rate 58 → 55, inside the noise of one run.
- MiniMax: cost −28% (interval 11 to 41%), the same mechanism, smaller in absolute terms because it was already carrying a third of the context.
- The stop rule was not obeyed by either model. The tail after the last edit halved and is still 36 to 48% of spend. That one belongs in the harness, enforced, not in the prompt.
So a third of the gap is a prompt away. The rest is the model. What a prompt can buy.
What this does not say
It does not rank the models in general: two harnesses, one plain prompt each, 78 django-heavy instances, one run per cell, provider pinned on OpenRouter. The 78 are the pilot's dispatch-order subset of Verified, not a random sample. A different task mix, a different harness prompt, or a provider whose cache pricing differs would move the ratio.
What it does say: a price sheet is a poor predictor of an agent's bill; a resolve rate cannot separate two models whose token behaviour differs by a factor of seven; and you need a per-call ledger under an enforced set of limits to see any of this at all. Every "cheap" model deserves the same 78-problem run before anyone believes the price sheet.
The record
Four runs. Each is replayable turn by turn in the Explorer and downloadable as a bundle with every patch, trajectory, per-call ledger row and eval report:
- Laguna · OpenCode: Explorer · bundle
- MiniMax · OpenCode: Explorer · bundle
- Laguna · mini-swe-agent: Explorer · bundle
- MiniMax · mini-swe-agent: Explorer · bundle
The framework that produced them is open source: agent-evals, and how every number on this site is made.
- run
- 01788910552328915713-21ebff89 · 2026-09-08
- model
- openrouter/poolside/laguna-xs-2.1 (alias laguna-xs-2.1-opencode)
- harness
- OpenCode · CLI 1.18.18 (pinned in the harness image build at this framework commit)
- framework
- dd365d8ff1197276992135731e67d81b17fb6e94
- harness image
- sha256:5f326fc00f2decd4f0592b10ac0c36defe26cfed3ae27e0eb24d3804d2df944f
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- run
- 01788991399526532442-883d2571 · 2026-09-09
- model
- openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-opencode)
- harness
- OpenCode · CLI 1.18.18 (pinned in the harness image build at this framework commit)
- framework
- 04a187d444a3f917b8e8642301f45cc28ed4769c
- harness image
- sha256:b1963fc83e1be63c3b25ca7688fac05a5ed9e6a7518333c7fe3cf6da03880dbb
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- run
- 01788997363735762375-c40be1f7 · 2026-09-09
- model
- openrouter/poolside/laguna-xs-2.1 (alias laguna-xs-2.1-mini)
- harness
- mini-swe-agent · CLI 2.4.6 (pinned in the harness image build at this framework commit)
- framework
- 3baa2e761bc0653a340a8c161c47c9bdf1b1ffd5
- harness image
- sha256:ea3836a6187ae133a670a9591e0023ce6662636a6d2185e424e8dd6919298b7f
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- run
- 01788995535869021545-f8f44aeb · 2026-09-09
- model
- openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-mini)
- harness
- mini-swe-agent · CLI 2.4.6 (pinned in the harness image build at this framework commit)
- framework
- 04a187d444a3f917b8e8642301f45cc28ed4769c
- harness image
- sha256:b1963fc83e1be63c3b25ca7688fac05a5ed9e6a7518333c7fe3cf6da03880dbb
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs