Cheaper per token is not cheaper per task
Two models of similar size and context length. Same harness, same system prompt, same tools, same 78 SWE-bench Verified problems, same limits. Repeated on a second harness as a check. This page is the data; the argument is written up in Cheaper per token is not cheaper per task.
The price sheet and the bill
| Laguna xs 2.1 | MiniMax M2.5 | Laguna is | |
|---|---|---|---|
| price per million tokens | |||
| uncached input | $0.06 | $0.30 | 5× cheaper |
| cached input | $0.03 | $0.03 | the same |
| output | $0.12 | $1.20 | 10× cheaper |
| the bill on OpenCode, 78 problems | |||
| resolved | 58 of 78 | 59 of 78 | the same, within the interval |
| inference cost | $15.45 | $3.49 | 4.4× more expensive |
| cost per resolved instance | $0.266 | $0.059 | 4.4× |
| the bill on mini-swe-agent, same 78 | |||
| resolved | 55 of 78 | 58 of 78 | the same, within the interval |
| inference cost | $15.55 | $4.37 | 3.6× more expensive |
| cost per resolved instance | $0.283 | $0.075 | 3.6× |
Cost per resolved instance, by model and harness
four runs · each model on each harness · same 78 instances, same limits, no prompt preset · lower is better
Inference spend only, from the gateway's own metering (compute cost is reported separately on the Cost page). Sorted cheapest first. A missing bar is an unmeasured run, not a free one.
Why: the bill is context × calls, and the discount is on neither
On OpenCode, Laguna made 2.6× the calls and carried 2.9× the context on each, so it sent 7.5× the prompt tokens for the same outcome:
| per 78 attempts, OpenCode | Laguna xs 2.1 | MiniMax M2.5 | ratio |
|---|---|---|---|
| LLM calls (median per attempt) | 6,514 (70) | 2,527 (23) | 2.6× |
| prompt tokens per call | 76.9K | 26.4K | 2.9× |
| prompt tokens in total | 501M | 67M | 7.5× |
| share of prompt tokens that are cache reads | 98.5% | 96.1% | — |
| output tokens | 1.75M | 0.64M | 2.7× |
Almost all of that context is cache reads, and cache reads are the one token kind priced the same on both models. Splitting each bill by token kind at the list prices above reproduces the metered total within one percent:
| share of the bill | uncached input | cached input | output |
|---|---|---|---|
| Laguna · OpenCode ($15.45) | 3% | 96% | 1% |
| MiniMax · OpenCode ($3.49) | 22% | 55% | 22% |
| Laguna · mini-swe-agent ($15.55) | 2% | 96% | 2% |
| MiniMax · mini-swe-agent ($4.37) | 27% | 49% | 24% |
So the 5× and 10× discounts apply to the four percent of Laguna's bill that is new text. The other ninety-six percent is a large conversation re-sent on every one of 2.6× as many calls, at a price Laguna does not discount. Where those calls go, whole-file reads, re-reads of unchanged files and repeated test runs, is on the model behaviour page.
The harness axis, for scale
On the same model, swapping OpenCode for mini-swe-agent moved MiniMax's bill from $3.49 to $4.37 and Laguna's from $15.45 to $15.55: a quarter at most. Swapping the model moved the bill by a factor of four on either harness. The pattern is the model's, and the second harness is the check that it is not an OpenCode artefact.
What this does and does not show
- On this subset the price sheet is a poor predictor of the bill, and resolve rate cannot separate two models whose behaviour differs by a factor of seven in tokens.
- It does not rank the models in general: two harnesses, one plain prompt each, 78 django-heavy instances, one run per cell, provider pinned on OpenRouter.
- Both models solve the same problems and fail the same ones: 55 of 78 resolved by both and 16 by neither on OpenCode; 51 and 16 on mini-swe-agent.
All the numbers
| harness | model | attempted | gradeable | resolved | rate / attempted | rate / gradeable | cost | $ / resolved | tokens in / out |
|---|---|---|---|---|---|---|---|---|---|
| OpenCode 21ebff89 | openrouter/poolside/laguna-xs-2.1 | 78 | 78 | 58 | 74.4% [63.7, 82.7] | 74.4% [63.7, 82.7] | $15.45 | $0.266 | 500.8M / 1.8M |
| mini-swe-agent c40be1f7 | openrouter/poolside/laguna-xs-2.1 | 78 | 78 | 55 | 70.5% [59.6, 79.5] | 70.5% [59.6, 79.5] | $15.55 | $0.283 | 501.7M / 3.0M |
| OpenCode 883d2571 | openrouter/minimax/minimax-m2.5 | 78 | 78 | 59 | 75.6% [65.1, 83.8] | 75.6% [65.1, 83.8] | $3.49 | $0.059 | 66.7M / 640k |
| mini-swe-agent f8f44aeb | openrouter/minimax/minimax-m2.5 | 78 | 78 | 58 | 74.4% [63.7, 82.7] | 74.4% [63.7, 82.7] | $4.37 | $0.075 | 75.2M / 865k |
| metric (78 paired, plain prompts) | Laguna · OpenCode | MiniMax · OpenCode | Laguna · mini-swe-agent | MiniMax · mini-swe-agent |
|---|---|---|---|---|
| cost | $15.45 | $3.49 | $15.55 | $4.37 |
| resolved | 58 (74.4%, CI 63.7–82.7) | 59 (75.6%, CI 65.1–83.8) | 55 (70.5%) | 58 (74.4%, one empty) |
| cost per resolved | $0.266 | $0.059 | $0.283 | $0.075 |
| ledger calls (median per attempt) | 6,910 (72) | 3,090 (27.5) | 8,738 (97.5) | 3,670 (38.5) |
| prompt tokens | 501M | 67M | 502M | 75M |
| prompt tokens per call | 72.5K | 21.6K | 57.4K | 20.5K |
| cached share of prompt | 98.5% | 96.1% | 99.1% | 94.9% |
| output tokens | 1.75M | 0.64M | 3.01M | 0.87M |
| median agent seconds | 447 | 116 | 673 | 164 |
| spend after the last edit | $6.73 (44%) | $1.92 (55%) | — | — |
| bill by token kind | uncached input | cached input | output | total (metered) |
|---|---|---|---|---|
| Laguna · OpenCode | 7.4M tokens · $0.44 | 493.4M · $14.80 | 1.75M · $0.21 | $15.45 |
| MiniMax · OpenCode | 2.6M · $0.78 | 64.1M · $1.92 | 0.64M · $0.77 | $3.49 |
| Laguna · mini-swe-agent | 4.7M · $0.28 | 497.0M · $14.91 | 3.01M · $0.36 | $15.55 |
| MiniMax · mini-swe-agent | 3.8M · $1.15 | 71.4M · $2.14 | 0.87M · $1.04 | $4.37 |
Ledger calls include refusals and retries, so they run higher than the trajectory-counted calls
in the "why" table. The cost column is always the metered ledger; the by-kind split is derived
from per-call token counts at the list prices. Verdicts OpenCode → mini-swe-agent on MiniMax: 55
both resolved, 15 both unresolved, 4 lost, 3 gained, one empty patch.
- run
- 01788910552328915713-21ebff89 · 2026-09-08
- model
- openrouter/poolside/laguna-xs-2.1 (alias laguna-xs-2.1-opencode)
- harness
- OpenCode · CLI 1.18.18 (pinned in the harness image build at this framework commit)
- framework
- dd365d8ff1197276992135731e67d81b17fb6e94
- harness image
- sha256:5f326fc00f2decd4f0592b10ac0c36defe26cfed3ae27e0eb24d3804d2df944f
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- run
- 01788991399526532442-883d2571 · 2026-09-09
- model
- openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-opencode)
- harness
- OpenCode · CLI 1.18.18 (pinned in the harness image build at this framework commit)
- framework
- 04a187d444a3f917b8e8642301f45cc28ed4769c
- harness image
- sha256:b1963fc83e1be63c3b25ca7688fac05a5ed9e6a7518333c7fe3cf6da03880dbb
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- run
- 01788997363735762375-c40be1f7 · 2026-09-09
- model
- openrouter/poolside/laguna-xs-2.1 (alias laguna-xs-2.1-mini)
- harness
- mini-swe-agent · CLI 2.4.6 (pinned in the harness image build at this framework commit)
- framework
- 3baa2e761bc0653a340a8c161c47c9bdf1b1ffd5
- harness image
- sha256:ea3836a6187ae133a670a9591e0023ce6662636a6d2185e424e8dd6919298b7f
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs
- run
- 01788995535869021545-f8f44aeb · 2026-09-09
- model
- openrouter/minimax/minimax-m2.5 (alias minimax-m2.5-mini)
- harness
- mini-swe-agent · CLI 2.4.6 (pinned in the harness image build at this framework commit)
- framework
- 04a187d444a3f917b8e8642301f45cc28ed4769c
- harness image
- sha256:b1963fc83e1be63c3b25ca7688fac05a5ed9e6a7518333c7fe3cf6da03880dbb
- benchmark
- SWE-bench/SWE-bench_Verified @ 78f471bf655a · swebench 5.0.2 · images SWE-bench_Verified-78f471bf655a.json
- network
- isolated
- explore
- this run in the Explorer: every instance, its trajectory and patch, every inference call, the judge’s verdicts
- download
- this run's artifacts (patches, trajectories, call ledgers, eval reports, judge passes) · preds.jsonl · all runs