Method
Where every date, price and score comes from. The index is built by forward-deployed AI engineers who move client agents from one model to the next. We have no model to sell, so the only thing the index has to be is right.
How we collect
Every lifecycle date, replacement, redirect and price comes from the provider or platform that sets it: their model-list APIs, deprecation pages, pricing pages and change feeds. Open model catalogs and news only tell us where to look. They are never the source of a fact.
Facts are kept per platform, because platforms disagree. The same model often shuts down on a cloud months before or after its maker’s own API, and some platforms quietly send an old model name to a newer model. Each fact names its source and shows the date we last checked it.
Firm dates and floors
A firm date is an announced shutdown. A floor is a provider’s “not sooner than” date: the model can stay longer, never shorter. The index labels the two apart everywhere, including in the calendar file Check my stack downloads.
How we verify
- Each date, replacement, redirect and price is matched to the exact sentence or table row it came from.
- Where a provider allows it, we keep a saved copy of the page, and the build fails if a quoted line is not on it.
- Pages that bar automated reading are read by a person.
- A person reviews every change to the record before it publishes.
- Prices keep their history: an earlier price stays on the record with the dates it applied.
How we measure agent work
Benchmarks rank general intelligence. The index also measures what an agent team needs to know before a model change: whether the task got done, what a finished task cost, whether the agent broke a policy or a permission, and whether it succeeds every time or only sometimes.
The core suite is 5 workloads and 280 tasks, scored by fixed checks rather than a model acting as judge. Part of the suite is private, so results cannot be tuned to the test.
| Workload | Tasks | What it checks |
|---|---|---|
| Tool calling over MCPCore | 60 | Multi-turn tasks where the agent chooses and calls tools served over the Model Context Protocol, with exact checks on the calls and the final state. |
| Customer support with policyCore | 60 | Airline, retail and telecom conversations where the agent must follow a written policy while a simulated customer pushes back. |
| Structured extractionCore | 60 | Business documents turned into typed records, scored field by field. |
| Tenant safety and permissionsCore | 60 | Requests that tempt the agent across a tenant boundary, instructions hidden inside tool results, and actions that need approval or stay under a limit. |
| Multi-app tasksCore | 40 | Tasks that span several apps, with checks that also catch unintended side effects. |
| Coding and long-running workDeep, for larger models | 30 | Repairs and builds that take many steps in a sandbox, checked by the project's own tests. |
Same tasks, same day
A replacement runs the same tasks as the model it replaces, through the same harness, on the same day. Every task runs at least three times. Each task gets its own verdict against the predecessor: reliably better, reliably worse, unchanged or inconclusive, with a correction so chance differences do not count as regressions.
What each result shows
- Tasks solved, with a confidence range. Models whose ranges overlap share a rank on the leaderboard.
- Reliability: the share of tasks solved in every run.
- Control: policy and permission violations per 100 tasks.
- Cost per finished task, failed attempts and retries included, and tokens per task, which shows a tokenizer change before the invoice does.
- Flip rate: the share of tasks whose outcome changed against the predecessor, net of the predecessor’s own run-to-run flips.
- The suite version, task and run counts, and the date it was measured.
Every measured result is reviewed before it publishes. Speed comes from fixed prompts sent to each platform, reported as medians with the sample count and date.
No model to sell
TeqEngine does not resell models or platforms, and no provider pays for placement, position or coverage in the index. Providers can ask us to correct a factual error. They have no say in results, rankings or what we write about a change.
When a client asks which model to move to, we answer with their own tasks, not with the leaderboard. The index shows the public picture; the decision for a given agent comes from that agent’s cases.
Corrections
Found a wrong date, a missing replacement or a stale price? Tell us which page and which fact, with a link to the provider’s source. We check it against the primary source and correct the record. The corrected fact shows its new checked date.
Data credits and licenses
Third-party data appears with its publisher’s name and license wherever it is shown, and keeps that license in the data file.
| Source | What the index uses | License and credit |
|---|---|---|
| Epoch AI | Benchmark results Epoch AI ran itself, shown as intelligence scores | CC BY 4.0Benchmark data: Epoch AI, licensed CC BY 4.0 |
| Arena | Human-preference ratings, shown as intelligence scores | CC BY 4.0Human-preference ratings: Arena, licensed CC BY 4.0 |
| OpenRouter | Public usage data, when the index shows it | CC BY 4.0Usage data: OpenRouter, licensed CC BY 4.0 |
| Model providers and platforms | Release, deprecation and shutdown dates, replacements, redirects, specs and list prices | Facts, cited to the provider's own pageEach fact links to its page and shows when it was checked |
Using the index’s data
The index’s own data is free to cite, copy and build on under CC BY 4.0. Download it as data.json, or follow changes in the Atom change feed. Credit it as:
Credit line
Source: TeqEngine Model Change Index (teqengine.ai/model-change-index)