The method
The evidence expires first
OpenAI stopped reporting SWE-bench. One paraphrase reorders model rankings 63% of the time. How to weight decision criteria by how fast each one decays.
In February 2026, the lab with the most to gain from a high SWE-bench score stopped reporting one.
OpenAI audited the problems its models kept failing: "We audited a 27.6% subset of the dataset that models often failed to solve and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submissions." Improvements on the benchmark "no longer reflect meaningful improvements in models' real-world software development abilities. Instead, they increasingly reflect how much the model was exposed to the benchmark at training time."
Every model choice made in 2025 on that number was partly measuring memorization. The prompts, tool schemas, retry logic and eval harness are still in production, resting on a number its publisher has disowned.
The evidence expired faster than the commitments it created. "Run your own evals" is the standard advice, but it skips the harder question: when you know the facts will rot, which ones still deserve weight, and which should you refuse to score at all?
The scoreboard has a half-life
In February 2026, FAIR, EPFL and AI2 published Brittlebench, which applies semantics-preserving perturbations: the same question reworded. Performance degrades "as much as 12%," those perturbations "can account for up to half of the performance variance," and "even a single perturbation alters the relative ranking of models in 63% of cases."
That is worse than decay: a score that unstable never had a half-life, and it is the number everybody weights first.
The ruler gets re-cut too. Artificial Analysis dropped a benchmark from its Intelligence Index in June 2026 — "Removed IFBench due to saturation" — and seven weeks later re-graded the same work with a different judge model, moving one model 2.7 points.
- Rankings reordered by one paraphrase
- 63%
- Brittlebench, February 2026. A semantics-preserving rewrite changes the relative ranking of some pair of models about two times in three.
- SWE-bench Verified problems with flawed tests
- 59.4%
- Of the 27.6% subset OpenAI audited — the problems its models kept failing. Tests that reject functionally correct submissions.
- Movement from swapping the judge model
- 2.7 pts
- Artificial Analysis re-graded the same work seven weeks after shipping v4.1. Nothing about the models changed.
Careful independent work fares no better. In August 2026 the author of the Ante harness matched DeepSeek's claimed 82.7% on Terminal-Bench 2.1 exactly, 368 successful trials out of 445, every trial record published. Within a day a Terminal-Bench enthusiast opened the job: "It looks like these were done by inflating the the timeout, which makes the score invalid." Runs marked successful had lasted 2h14m and 5h54m against an official task limit of 3600 seconds.
That is as good as public benchmark evidence gets. The number never decayed; it was wrong on arrival, and nobody could tell until a stranger read the logs.
The name in your config is not a spec
Pick a model and pin the version string. You have pinned less than you think.
On 23 April 2026, Anthropic published a postmortem covering six weeks of quality complaints. Three unrelated changes, none of them a model: default reasoning effort cut from high to medium on 4 March, restored 7 April; a caching optimization on 26 March that cleared older thinking from idle sessions, fixed 10 April; a system-prompt instruction to reduce verbosity, shipped 16 April and reverted on the 20th. On that last one the company reports "a 3% drop for both Opus 4.6 and 4.7" on one of its evaluations.
The identifiers never moved. The system behind them did, and the people who noticed were treated as unreliable narrators until the postmortem landed.
The drift is industry-wide. In a paper revised in August 2026, Chauvin and colleagues probed "131 endpoints covering 78 models across 34 providers," then monitored 53 of them every 24 hours for 23 days: eight showed persistent change under their own names. Watching an endpoint hourly for a year costs $0.52, and almost nobody pays it.
The economics move under the same names. In April 2026 a Claude Code user reconstructed his prompt-cache history from 119,866 logged API calls. From 1 February to 5 March, every cache write used the one-hour tier; on 6 March the five-minute tier reappeared, and by 8 March outnumbered it five to one. His arithmetic put the overpayment at 17.1%. Anthropic disputed the conclusion rather than the detective work: its engineer Jarred Sumner said the five-minute default made Claude Code cheaper.
"A meaningful share of Claude Code's requests are one-shot calls where the cached context is used once and not revisited."
Both can be right, which is the problem. One server-side change bills one usage pattern up and another down, and nobody published which one you are. The Opus 4.6 to 4.7 tokenizer change landed the same way. Identical text produced about 45% more tokens, "more than 2x for small prompts," while Artificial Analysis's suite ran input costs up $800 and output costs down $1,400, net cheaper. The rate card never moved. Your bill did, in a direction nobody benchmarked, including you.
What actually moves slowly
The durable criteria are the boring ones, unscored in every comparison doc.
Deprecation notice is the clearest. Anthropic commits to "at least 60 days' notice before model retirement for publicly released models," and in 2026 landed within a day or two of that floor every time: Opus 4.1 deprecated 5 June and retired 5 August, Sonnet 4 and Opus 4 given 14 April to 15 June. OpenAI's published policy is "at least 6 months" for generally available models, three months for specialized variants, and for previews "much shorter notice, such as 2 weeks."
| Vendor and tier | Notice |
|---|---|
| OpenAI — generally available models | ≥ 6 months |
| OpenAI — specialized variants | ≥ 3 months |
| Anthropic — public models | ≥ 60 days |
| OpenAI — preview models | ~2 weeks |
Sixty days is one sprint; a hundred and eighty is a planning cycle.
The API surface moves slower than the models, but when it moves it deletes things. temperature, top_p and top_k now return a 400 error on Claude Opus 4.7 and later, and Google's documentation says the same is coming: "In future model generations, supplying these parameters returns an HTTP 400 error. Remove these parameters from all requests."
Then there is the criterion nobody scores: how exposed your vendor is to something that has nothing to do with you. On 12 June 2026 the US government applied export controls to Claude Fable 5 and Mythos 5; the order took immediate effect and nationality checks were unreliable, so Anthropic pulled both models for everyone. They came back on 1 July, with a classifier that blocks the flagged technique "in over 99% of cases" and "comes at the cost of flagging benign requests more often during routine coding and debugging tasks."
Nineteen days of nothing, then a permanent tax on false positives. The outage was sudden. The exposure behind it was a stable property of that vendor, knowable in advance and absent from every model comparison written that quarter.
The slowest-moving thing here is the part you built. Armin Ronacher, in July 2026, on tuning agent harnesses to specific models: "The problem with hyper targeting harnesses to models is that you end up locking yourself quite quickly into special behaviors of models, and you make your sessions non transferrable." Chase a fast-decaying criterion hard enough and it manufactures a slow-decaying cost.
The best argument against all of this
The top model on ProgramBench, as of 3 August 2026, resolves 1.0%. Meta Superintelligence Labs built it with Stanford and Harvard out of 200 tasks and more than 248,000 behavioral tests. Capability is nowhere near a ceiling.
METR's January 2026 update puts the doubling time of the 50% time horizon at 88.6 days measured from 2024, with Claude Opus 4.5 handling tasks that take humans about 320 minutes. Fast decay and fast growth are the same fact seen from two sides: if capability doubles roughly every quarter, today's rank is worthless in ninety days, and the model available in ninety days is twice as capable, which is worth more than the migration you were dreading. On that reading, chasing the frontier is the correct response to a steep curve.
Practitioners put it more bluntly. From a 2,107-point thread in July 2026: "don't try to customize anything, just use it vanilla, and use the best model you can afford. No AGENTS.md, no special subagents or roles... Making a huge custom setup is so 2025." Switching may be cheap, and buyers behave as if it were: a Dataiku-commissioned Harris Poll of 600 enterprise CIOs, fielded between December 2025 and early January 2026, found "Fifty-five percent of CIOs say they have already switched LLMs at least once, with cost reduction as the primary driver."
If switching is nearly free, an elaborate comparison is theatre. If it is expensive, it is expensive because of things that are not the benchmark score: tool schemas, eval suite, retry logic, the finance conversation. A developer who moved 47 multi-step workflows from Claude to a local Qwen model over two weeks in June 2026 reported matching quality on planning, and a tool-call format error rate of 12% against Claude's 0.5%. The gap that mattered was in the interface, not the intelligence.
The third branch is the one worth taking: capability per dollar on your own workload. It decays in months rather than weeks, which earns it real weight, and it is the only version of "capability" you can defend in a room. It is not a leaderboard position. Nothing in the counter-case rescues the published rank you gave 40% to.
Put the half-life next to the weight
The fix is mechanical. Write down your commitment horizon: the honest interval before you would realistically re-pick. For most teams shipping on an API it is somewhere between one and four quarters.
Then, next to each criterion's weight, write its half-life: how long you expect that fact to survive.
- Benchmark rank: weeksOne paraphrase reorders the ranking 63% of the time, and the index gets re-cut twice a year.
- Behaviour under a fixed name: weeksEight of 53 endpoints drifted in 23 days, with the version string unchanged.
- Price per token: monthsIt moves in both directions. OpenAI cut GPT-5.6 Luna by 80% on 30 July 2026.
- Effective cost per task: monthsNot the same fact as price. A tokenizer change re-prices you without touching the rate card.
- Latency and throughput: monthsThe one fast-decaying criterion worth scoring, because re-measuring costs an afternoon.
- Deprecation notice: yearsPublished policy. Still true next year, and it decides how bad a forced migration gets.
- API and parameter surface: quarters to yearsSlow, but it deletes things when it moves. Ask the people who built on temperature.
- Data-retention terms: contract lengthWhy a regulated team can use one vendor and not another, regardless of any score.
- Your own eval suite: as long as you maintain itThe only instrument that measures your workload rather than someone else's.
- Your tool and harness shape: as long as the codebase livesThe real switching cost, and the only one on this list you control.
Discount each weight by how much of that criterion survives your horizon. A fact with a half-life of weeks, against a commitment of quarters, contributes close to nothing and should be scored close to nothing. None of this demotes capability: if one model is the only thing that clears your task, that is a constraint, filtered before scoring rather than weighted inside it.
Then run the flip test. Re-score the decision as if it were ninety days from now: leaderboard reshuffled, one vendor's model unavailable for a fortnight, your provider's defaults changed. If the winner changes, you decided between two guesses about the next quarter, and you should go get a better fact.
Which is the real reason to run your own evals: it is the only criterion on this list whose half-life you set yourself.
If the winner holds, the decision was never about the score. It was about whatever was going to outlive it: how much of your system you are willing to rebuild when the answer changes. You make that choice long before any benchmark comes out.