The accuracy figure in the demo was measured on somebody else’s data. Build your own test set before you sign.

The supplier's slide says 94% accurate. Somebody on your side wrote it down, it went into the business case, and the business case went to the board. Nobody asked the obvious follow-up: 94% of what?

Not of your documents. Not of your customers' emails. Not of the awkward cases your team handles every Thursday afternoon. It was measured on whatever the supplier had to hand, and your data was not in it.

A number without a test set is a claim, not a measurement

An accuracy figure means something only when you know what it was measured against. Change the examples and the number changes, sometimes beyond recognition.

The clearest recent case came from the US Federal Trade Commission. A company sold an AI content detector as "98 percent" accurate. The FTC alleged that the model had only been trained to classify academic writing, and that on general-purpose content independent testing put it at 53 percent:

The FTC alleged that the AI model powering the AI Content Detector was only trained or fine-tuned to effectively classify academic content, despite the company's claims that it was developed using a wide range of materials.

For a yes-or-no question, 53% is a coin toss. The 98% was probably real, on the material it was tested on. It just was not the material the customers had.

That is the pattern to worry about. The supplier is rarely lying. They are reporting a true number about a different problem.

The gap is bigger on real business work

Public benchmarks tell you even less. Salesforce's research team built a benchmark from realistic sales, service and quoting tasks, precisely because the standard ones did not reflect business use. Their results were sobering: leading AI agents managed around 58% success on single-step tasks, falling to about 35% once the task ran over several turns of conversation. They also found "near-zero inherent confidentiality awareness".

Those were the best models available, on tasks designed to look like ordinary CRM work. Your process has quirks no benchmark has seen.

Most teams do not test properly either

You might assume the people building these systems all have this covered. LangChain's 2026 State of AI Agents survey, 1,340 practitioners surveyed in late 2025, found that only 52% run offline evaluations on test sets, and 29.5% do not evaluate at all. Among teams that already have agents in production, the figure not evaluating is still 22.8%.

Put plainly: roughly one in five AI systems already live has no systematic test of whether its answers are right. The organisation using it is relying on nobody complaining.

What is a test set, and who should build it?

A test set is a collection of real examples from your own work, each with the answer your best person would have given, agreed before the AI sees any of them. Somewhere between 100 and 300 examples is enough for most business processes. It is the only way to turn "94% accurate" into a number that applies to you.

Here is the part suppliers do not volunteer: you have to build it, not them.

It has to come from your side because only your people know what a correct answer looks like, and because a supplier who picks the test examples also picks the result. That is not an accusation. It is the same reason you do not let a firm audit its own accounts. We covered the incentive in more detail in why your AI will cheat if cheating is easier, and it applies to people as much as to models.

It is also tedious. Someone who knows the process has to sit with a spreadsheet for two or three days and write down the right answers. That is the real reason it rarely happens. It is dull, it lands on your busiest person, and nobody quoting for the build will ask for it, because it is the one thing that could make their demo look worse.

What it looks like when it is done

When we built a system that turns bank statements into structured transaction data, a generic extraction pass looked excellent on the first sample. Tested against real statements from real banks, with their different column layouts, multi-line descriptions and running balances in different places, it fell apart. That result is why the finished system extracts differently per bank, and why it still works.

On a support triage system for another client, the measure was the categories the support team already used, on the mail they actually received. That gave a number the client could check for themselves, 92% categorisation accuracy, and more usefully it showed which 8% went wrong and why. That is a very different conversation from "the model got it wrong".

In both cases the test set was not paperwork. It changed what we built.

What to do before you sign

Ask what the accuracy figure was measured on. How many examples, whose data, and who decided the right answers. If the answer is vague, treat the number as marketing.

Build a test set from your own work. A hundred or more real cases, including the awkward ones, with the right answers written down by the people who do the job. Keep it to yourself until the supplier's system is ready to be tested.

Agree the pass mark before anyone runs it. Decide what accuracy you need and what a wrong answer costs. An 85% system that flags its uncertain cases for a person may be worth more than a 95% system that is confidently wrong on the rest.

Keep the test set after go-live. It is how you will know whether the system still works after the underlying model changes, which it will. This is the same set you would use to check whether your data can carry the project in the first place.

Make it a condition of the proof of concept. A properly scoped proof of concept has success criteria agreed before it starts. A test set is what makes those criteria mean anything.

The uncomfortable version

If you cannot write down a hundred examples of the right answer, you are not ready to buy an AI system. You are not yet sure what you want it to do.

That is fixable, and it is cheaper to fix now than after the build. Our four-week proof of concept starts by building that test set with your team, runs the prototype against it, and reports the score, including when the score says not to proceed. It is £18,000 at a fixed price, and you keep the test set whatever the verdict. You can see exactly what is handed over on what you get.