Our model router kept picking the expensive model. A $25 control run tied it.
· the plori team
TL;DR. We replaced our model router's hand-written capability scores with measured benchmark indices, and the router immediately converged on the highest-scoring model for agentic work: 47 of 47 routed decisions on a 50-task coding benchmark went to kimi-k3, at 33x the input price of the cheapest candidate. So we ran the control: the same 50 tasks pinned to that cheapest model, deepseek-v4-flash. Compared on a same-harness basis, the result was a tie, 34 of 50 resolved each way, and the pinned run cost 44% of the routed run's model spend. A benchmark index gap that looks decisive on a scoreboard did not transfer to our workload. The calibration a router actually needs is outcome data from its own traffic.
Plori runs coding agents on cloud computers. Each agent's model calls go through a router: a small, fast judge model reads the shape of the request and picks a serving model from a price-ordered candidate list. The instruction to the judge is deliberately conservative. Start from the cheapest candidate. Step up only when a costlier candidate scores meaningfully higher on the dimension the task actually needs, and the task is demanding enough for the difference to matter.
That design has two failure modes, and we have now hit both.
First failure: hand-written scores
The first version of the candidate list carried hand-written capability scores, 1 to 5, written by us on vibes and partial evidence. Those scores rated one mid-price model a full tier above deepseek-v4-flash at coding. Independent measurement disagreed: on Artificial Analysis's coding index the two models were within half a point of each other, and the "weaker" one cost, at the time, about a third as much. The judge trusted our numbers and had converged on the mid-price model for essentially all coding traffic.
Hand-written scores fail in a predictable direction. They encode the reputation a model had when someone last paid attention, and reputations lag. So we deleted the prose and the 1-to-5 scores and replaced them with measured composite indices (intelligence, coding, and agentic tool use, each 0 to 100) from Artificial Analysis, snapshotted into the router's catalog with the variant matched to the reasoning effort we actually serve.
Second failure: measured scores, faithfully followed
With measured indices in place we ran a 50-instance subset of SWE-bench Verified through the platform, router live, nothing pinned. The subset is stratified by the dataset's difficulty labels and selected by a recorded seed, so the draw is reproducible and was not chosen after seeing results.
The judge chose the same model on 47 of 47 decisions it completed: kimi-k3, the candidate with the pool's highest agentic index. Two more tasks fell back to a static default when the judge timed out, and one task never reached a model call because of a platform fault. A monoculture, again, except this time every step of it was defensible. These were real multi-step agentic coding tasks. The relevant index really was several points higher than every cheaper candidate's. The judge did exactly what we told it to do.
The catch is the price column. Public per-token prices at the time of the run:
| Candidate | Input $/M | Output $/M | Agentic index* |
|---|---|---|---|
| deepseek-v4-flash | $0.09 | $0.18 | 45.7 |
| kimi-k3 | $3.00 | $15.00 | 50.1 |
*Artificial Analysis agentic composite, 0 to 100, snapshot 2026-08-05.
A 4.4-point index gap was buying a 33x input-price step on every routed task. Whether that trade is good depends entirely on whether the gap transfers to real outcomes, and nothing in a composite index promises that it does. Benchmark composites are model-intrinsic; your workload is not.
The control run
The experiment that settles it is cheap and boring: run the same 50 instances again with the model pinned to deepseek-v4-flash, router bypassed entirely. We verified from the decision log that the pinned round produced zero routing decisions. Concurrency, per-task time limit, prompts: identical. The harness itself had picked up two recovery mechanisms between the rounds; the honesty note below deals with that. A few tasks in each round (one routed, three pinned) hit platform faults unrelated to either model and were re-run cleanly; re-runs count normally in both columns.
| Basis | Routed (kimi-k3 on 47/50) | Pinned (deepseek-v4-flash) |
|---|---|---|
| Resolved, same-harness basis | 34 / 50 | 34 / 50 |
| Model spend | $39.62 | $17.30 |
| Total billed (models + compute) | $45.27 | $25.38 |
| Total tokens | 27.2M | 33.9M |
| Per-token blended cost | ~$1.46/M | ~$0.51/M |
A tie on resolve rate, 34 each way. The pinned model burned 24% more tokens to get there, which is what a weaker model grinding harder looks like, and it still finished at 44% of the model spend and 56% of the total bill.
The honesty note on the "same-harness basis" row. The pinned round ran later the same day, after a harness improvement had landed, and two new recovery mechanisms (reading a partial patch off the agent's disk after a timeout, and a short mechanical follow-up run that extracts the diff when the agent never wrote the patch file) recovered 5 additional resolved instances for the pinned round. Its headline number on the newer harness was 39 of 50. Those 5 are excluded from the comparison above because the routed round predates the mechanisms and never got the same help. Splitting the bases is the difference between a model comparison and a harness changelog.
The failures were different in kind
The tie hides an interesting asymmetry. The two models did not fail the same way.
Twelve of kimi-k3's sixteen failures were wall-clock timeouts, all in one large repository whose test suite dominates the clock. Its patches, when it wrote them, were clean: 92% of the patches it produced passed the benchmark's official evaluation.
deepseek-v4-flash showed two failure modes kimi-k3 never did. Twice in 50 tasks, the upstream stream ended abnormally before a usable turn. And four times it wrote junk diffs, from just under 1MB to 13MB: the one small enough to read buried a one-file real change under 246 files of repository noise, and it failed to even apply. The other three were too large for the harness to read at all. A cheaper model is not just "the same but slower." It has its own tail.
None of that tail was big enough to move the total on this subset. It might be on yours, which is rather the point of running your own control.
What a router should actually trust
The uncomfortable summary: we replaced bad hand-written numbers with good measured numbers, the router followed them faithfully, and it still made a call that a $25 experiment showed bought nothing measurable. Not because the indices are wrong. They measure what they measure, on their benchmarks, at a serving configuration that may or may not match yours. The 4.4-point agentic gap is probably real. It just did not convert into resolved tasks on this workload at this time limit.
So we are changing the routing policy in one specific way: index gaps alone will no longer justify a price step. The router's catalog will gain a first-party column, per-model resolve rates measured on our own periodic control runs, and a step up the price ladder will have to be backed by an outcome gap, not a benchmark gap. Until that ships, the change is policy, not code: the judge stays on the cheapest capable candidate unless the pair has evidence. External indices still do the job they are actually good at: catching a model that is far off the pace, and ordering candidates for tasks we have no first-party data on yet.
If you run a model router, the experiment is worth stealing. Pin the cheap model. Run your real workload twice. It costs about as much as a nice lunch, and it turns a routing debate into a table.
Caveats
- One round per configuration, 50 tasks each. The tie is "no measurable difference on this workload," with roughly a one-to-two task noise band, not a claim that the models are equal in general.
- One workload family (repository-scale bug fixing under a 25-minute per-task limit). Different task shapes can absolutely justify the step up; that is what the per-pair evidence requirement is for.
- These are not leaderboard numbers. Our agent stack, prompts, and time limits are part of the measurement, and the subset is 50 of the dataset's 500 instances.
Quality indices courtesy of Artificial Analysis. Tasks from SWE-bench Verified. Prices are the public per-token rates at the time of the run.