Your supplier did not change anything. The model underneath your AI did.

Your claims-triage assistant has worked well for eight months. This week the team says it has started routing more cases to the wrong queue. You ask the supplier what they changed. They check, and the honest answer is: nothing.

They are telling the truth. The model underneath your system changed.

Most organisations that bought an AI system think of it as software they own: built once, tested once, stable until somebody edits it. A system built on a hosted model from OpenAI, Anthropic or Google is not that. It is a thin layer of your logic on top of something another company updates on its own schedule, and most of those updates are never announced to you.

Do AI models change after you deploy them?

Yes. The large model providers adjust the systems behind their APIs after release, through retraining, tuning, routing and infrastructure changes, and they do not always say so. A study published in August 2026 looked at nine model providers and seven companies that host models for others, and found that none published enough information for an outsider to confirm that the model being served was the one described in its documentation.

That is not a fringe finding. Two well-documented cases show what it looks like from the inside.

In April 2025 OpenAI shipped an update to the model behind ChatGPT that made it, in its own words, "overly supportive but disingenuous". It was rolled back within days after users noticed it endorsing bad decisions.

Between 5 August and mid-September 2025, three infrastructure bugs degraded some of Claude's answers. At the worst hour, 16% of requests to one model were affected. Anthropic's own write-up says its standard evaluations did not catch it, partly because the model often recovered from isolated mistakes.

Read the second one again. The company that builds the model, with full access to its own infrastructure, took weeks to work out that something was wrong. Neither case was malicious, and both providers published honest accounts. That is the point: this happens to careful companies, as a normal part of running these systems.

Why your supplier will not notice either

Here is the part nobody puts in the proposal.

Many AI systems are wired to a model name such as "the latest version of model X", because it is simpler and the supplier gets improvements for free. Some providers let you pin a fixed version instead, which means changes only reach you when you choose. Pinning is the obvious protection, and plenty of builds skip it.

Even pinned, you are not safe for long. Fixed versions are retired on the provider's timetable, usually within a year or two, as we covered in who owns your AI after the supplier leaves. At that point somebody has to move you to a new model, and the new model will behave differently in ways nobody has measured.

And nothing in a typical setup is watching for it. The dashboards show the system is up and answering quickly, which is not the same as watching what it says. A model that has started misreading one category of claim still returns a confident, well-formatted answer in 800 milliseconds. Every light stays green.

A 2026 paper on governing updates in the AI supply chain makes the same point from the engineering side: providers update hosted models without changing the version name, and the resulting drift breaks the specific things an application depends on, such as the format of an answer, its error handling, or a rule about what it must refuse.

So the general benchmark scores go up, and your claims queue gets worse.

What a model change looks like in practice

Take a firm using an AI system to read inbound supplier invoices and extract the payment terms. The provider improves the model. It now gives fuller, more helpful answers, and it has started writing "30 days from invoice date" where it used to write "30". The finance system expects a number. Nothing errors. The field is quietly left blank on a few hundred invoices before anyone in accounts payable asks why.

Or a customer service assistant that used to refuse to discuss pricing changes now answers helpfully, because the new model is more willing to help. Nobody tested for the refusal, because it was never a problem.

Neither is a disaster. Both are found by customers or staff, weeks late, rather than by a test on the day the change arrived.

What to ask for

You do not need to understand the model to protect yourself from it changing. You need four things, and you can ask your supplier for each of them in writing.

• A pinned model version. Ask which exact model version your system calls, and whether that can change without anybody deciding it should. If the answer is "it uses the latest", ask why.
• A test set built from your own cases. A few hundred real examples with the correct answers, including the awkward ones and the things it must refuse. This is the same test set we argued you should build before you sign. Its second job is catching change.
• A rule that the test set runs on every model change, and on a schedule even when nobody thinks anything has changed, with results you can see. A score that drops from 94% to 88% is a conversation. A score nobody ran is a surprise.
• A named owner for the model upgrade. When the pinned version is retired, who does the migration, who reruns the tests, and who signs off? If the contract ended at go-live, the answer is probably nobody.

A supplier who has thought about this will answer all four in a short email. A supplier who has not will tell you the model is very reliable.

Who checks once the build is finished

This is the gap our Design Authority service exists for. From £1,500 a month, we sit on your side of the table after go-live: we hold the test set, check that it runs, read the results when the model underneath changes, and tell you plainly whether the system you are paying for still does what you bought it to do. It stands alone, so it works whoever built the system.

The model will change. The question is whether you are told by your test set or by a customer.