All ten best-value models held their places this week; combining cheaper models matched top quality on code work for a ninth of the cost.
Every week Potion tests AI models on ten kinds of work to find which ones give the best value. A model makes the list when nothing else beats it on quality, speed and cost at once. This week all ten top picks stayed stable when re-checked. Three newly released models were tested and none earned a place. The standout finding: on code work, using a few cheap models together matched the best single model's quality for less than a ninth of its price.
Week 34 brought 325 newly listed models to screen and 3 to measure in full. None of the ten frontier picks moved. The most useful number this week: on code work, a combination of measured models scored 1.000 quality, matching the best single model, and cost more than eight times less per thousand requests.
This week's frontiers
One row per kind of work. Routed pick is the model Potion currently sends that work to. Stored quality is its exam score when it was measured in full, give or take the margin. Canary is this week's small re-check: a few fresh tasks, scored the same way, to catch a model that has got worse. Held means the re-check landed inside the margin.
| kind of work | verdict | routed pick | stored quality | canary |
|---|---|---|---|---|
| agentic-tool-use | held | or-gemini-flash | 0.893 ± 0.095 | 0.900 n=4 |
| classification | held | name withheld | 0.976 ± 0.033 | 1.000 n=4 |
| code-gen | held | name withheld | 0.979 ± 0.020 | 1.000 n=4 |
| code-review | held | or-gpt-mini | 0.900 ± 0.081 | 1.000 n=4 |
| creative | held | or-sonnet | 0.907 ± 0.032 | 0.875 n=4 |
| extraction | held | name withheld | 0.960 ± 0.018 | 1.000 n=4 |
| multi-step-reasoning | held | or-deepseek-v4-flash-0731 | 0.980 ± 0.039 | 1.000 n=4 |
| rag-answer | held | name withheld | 0.920 ± 0.076 | 0.750 n=4 |
| rewrite-edit | held | or-gpt-mini | 0.872 ± 0.059 | 0.850 n=4 |
| summarization | held | or-ling-3.0-flash | 0.893 ± 0.063 | 0.925 n=4 |
A canary is a small weekly re-check of four requests that catches if a model has collapsed. All ten frontier picks passed. The margin of error is how sure we are: or-gemini-flash on agentic-tool-use scored 0.900 this week against its stored score of 0.893, give or take 0.095. Classification and code-gen both hit 1.000 observed against stored scores of 0.976, give or take 0.033, and 0.979, give or take 0.020.
Auditions
An audition is a newly released model's first exam, on the kind of work it looks suited to. It earns a place only by beating the model already doing that work on quality or price.
An audition is a first exam for a newly released model. This week or-ling-2-6-flash tried the multilingual lane for rewrite-edit work and did not beat the model already there. Two small models, or-l3-lunaris-8b and or-nex-n2-mini, were measured on classification work and neither beat the incumbent.
- or-ling-2-6-flash · multilingual · rewrite-edit · did not beat the incumbent
- or-l3-lunaris-8b · small/cheap · classification · did not beat the incumbent
- or-nex-n2-mini · small/cheap · classification · did not beat the incumbent
Mixing
Mixing means using two or three cheaper models together in a particular way instead of one expensive one. We score combinations by replaying results we already have, so this costs nothing to explore. How a combination works is part of the product and is not published; what it achieves is.
A replay means re-scoring a combination from results we already have, without spending money on new requests. On code work, ninety items were replayed through a combination of measured models. The combination scored 1.000 quality and cost more than eight times less than the best single model. On structured output work, a combination scored 1.000 over eighty items and cost between four and eight times less.
- code work · cheaper-and-as-good · quality 1.000 · more than 8× cheaper · n=90
- structured output work · cheaper-and-as-good · quality 1.000 · between 4× and 8× cheaper · n=80
- writing work · cheaper-and-as-good · quality 0.979 · between 2× and 4× cheaper · n=14
- retrieval work · cheaper-and-as-good · quality 0.980 · between 2× and 4× cheaper · n=50
- reasoning work · cheaper-and-as-good · quality 0.993 · between 1.5× and 2× cheaper · n=14
- summarization · cheaper-and-as-good · quality 1.000 · 19% cheaper · n=14
- code work · frontier-candidate · quality 0.995 · under 1.5× cheaper · n=42
- writing work · frontier-candidate · quality 0.929 · under 1.5× cheaper · n=14
- extraction · frontier-candidate · quality 0.987 · -507% cheaper · n=74
What it means for you
The difference between the most expensive model and the cheapest one that passes your quality bar is where your budget goes. This week that gap was measured again across ten kinds of work. Routing each request to the cheapest model that scored high enough is how you keep the quality and avoid paying for performance you do not need.
How the numbers are made
How these numbers are made, in plain words. We sort requests into kinds of work: sorting text into categories, pulling fields out of documents, writing code, answering from a set of documents, and so on. For each kind we keep a private exam of tasks the models have never seen. Every model sits the same exam. Code is marked by running it; other answers are marked against a reference answer. A model's quality is its average mark, and because an exam is a sample we also give a margin of error: two models whose margins overlap are called a tie. Cost is what a thousand requests would cost at the provider's public prices. A frontier is the short list of models that are the best deal at their level of quality, meaning nothing else is both better and cheaper. Each week we re-check every model on that list with a few fresh tasks to catch any that have got worse, and we give newly released models the exam for the work they look suited to.
- Every quality figure is a mean over a retrieval-hostile suite with a bootstrap 95% interval; two points whose intervals overlap are reported as tied.
- Code-generation and code-review quality is scored by executing the code; other clusters are scored by a rubric against a reference answer.
- A canary is a small weekly sample (four items) against the stored measurement; it detects collapse, not one-point movement.
- Combinations of models are replayed from stored item-level results; agreement between models is modelled conservatively, so the combination figures understate rather than overstate.
- Free-tier listings are excluded from auditions: their quality is not stable enough to measure.
Numbers
10 canaries · 10 held · 0 moved · 0 inconclusive · 40 items graded · 325 listings screened · 3 measured · $2.01 spent
Questions
- What does frontier mean in AI model testing?
- A frontier is the short list of best-value models for one kind of work. A model makes the list when nothing else beats it on quality, cost and speed at the same time. A routing system picks from it based on your needs: cheapest above a quality floor, fastest, or highest quality.
- Did any AI model get worse this week?
- No. All ten frontier picks reproduced their stored quality within the margin of error on a four-item canary. The canary detects collapse, not small movements. Every observed score this week fell inside the 95 percent confidence interval of the stored measurement.
- How much can I save by routing to cheaper models?
- It depends on the kind of work. This week code work showed the widest gap: a combination of cheaper models matched the top quality for more than eight times less per thousand requests. Structured output work saved between four and eight times. Writing and retrieval work each saved between two and four times.
Glossary. A frontier is the short list of models that are the best deal at their level of quality: nothing else is both better and cheaper. A floor is the lowest exam score you are willing to accept. A margin (or interval) is how far the true score could sit from the measured one, because an exam is a sample. A canary is a small weekly re-check. See the docs and the evidence.
This issue was drafted by sending one request to Potion's own API, the same way a customer would. The receipt that came back: kind of work creative, strategy 07b4dc72, policy max_quality, 4862 tokens. We use what we sell.