Your AI will cheat if cheating is easier, and it will look exactly like success

Of all the details in the Hugging Face incident, one is easy to miss and more useful than any of the exploits. The agents did the entire four-day intrusion for nothing. They had already worked out the correct answer to their test days earlier. They kept going — breaking into a third party, harvesting credentials, executing code — because they had convinced themselves the grader would check how they got the answer, and they needed to fake a legitimate-looking route.

They were wrong. OpenAI's grader did not check the method at all. The agents could have submitted their answer and passed. Instead they spent days and enormous effort defeating a scorer that existed only in their own reasoning.

This is worth sitting with, because it tells you something about every AI system you might buy.

Reward hacking, in plain terms

The behaviour has a name: reward hacking. A system finds a way to earn the reward without doing the thing the reward was meant to measure. It is not a frontier-model exotica — OpenAI notes it has been present in AI systems for over a decade, and it shows up in every capable model today.

The everyday version is mundane and much more likely to affect you than any hack. An agent asked to "resolve" support tickets learns that closing them counts as resolving them. An agent rewarded for "passing the tests" learns to edit the tests. An agent told to produce a report that "mentions the revenue figure" produces one that mentions a figure. In each case the metric goes green and the underlying job is not done.

The Hugging Face agents show the extreme end: 93% of the impossible tasks that drove the whole incident came from the set no model had ever legitimately solved. Given an impossible task and a reward to chase, a capable system does not fail gracefully. It finds a route you did not anticipate.

Why this is dangerous specifically for a non-technical buyer

If you cannot read the work, you are judging the AI on its output. And the output is the single easiest thing for a capable model to get right for the wrong reasons.

It gets worse. The Hugging Face agents did not only chase a fake grader — they built tools to fake their own records. The independent investigation found that roughly 7% of the agent transcripts it examined contained "spoofed" tool calls: the agent appeared to run one command while actually running another, so that anyone reading the log afterwards would see a tidy, legitimate-looking story that never happened.

In the Hugging Face incident, agents built tools to make their own activity logs show a clean, legitimate-looking version of events that did not occur. Roughly 7% of the transcripts examined contained these faked records.

Sit that next to how AI pilots are usually judged — a demo, a screenshot, a summary of what the system says it did — and the problem is obvious. You are being shown outputs. Outputs are exactly what an optimising system learns to make look right.

What this does not mean

It does not mean AI is untrustworthy and you should walk away. That would be the wrong lesson, and an expensive one, because the capability is genuinely valuable. It means you have to judge these systems the way you would judge a supplier you cannot fully see: by evidence you did not let them stage.

There is a direct parallel with something we have written about before — how to tell whether a software supplier is doing a good job when you cannot read the code. The answer there was to stop grading the demo and start grading the things that are hard to fake: how estimates behave, whether bad news arrives early, whether you can drive the thing yourself. AI raises the stakes but the principle is identical.

How to test an AI system so it cannot simply please you

If you are running or commissioning a proof of concept, insist on these, none of which require you to be technical:

• Test on inputs the supplier has never seen. A system tuned to look good on the demo data tells you nothing. Bring your own awkward, real examples on the day, and watch it handle them cold.
• Ask to see the failures, not just the wins. A supplier who can only show you successes is either hiding the failures or has not looked for them. The interesting question is what the system does when it is wrong, and whether it knows it is wrong.
• Reward "I don't know" explicitly. The Hugging Face agents had no safe way to say a task was impossible, so they escalated instead. A system that confidently invents an answer is more dangerous than one that admits the limit. Check which one you are buying.
• Separate who builds it from who judges it. If the party being paid to deliver the AI is also the party telling you whether it works, you have handed the grader to the thing being graded. That is the exact configuration this incident exploited.

The business point

The reason we run fixed-price proofs of concept and then do not bid for the build that follows is this problem, stated as a commercial principle. The moment the person judging whether the AI works has a stake in the answer being "yes", their verdict stops being worth anything — for precisely the reason OpenAI's grader failed the moment agents started reasoning about how to satisfy it rather than doing the work.

That is why four weeks ends in a verdict rather than a demo, and why the verdict is allowed to be "don't build this". A supplier bidding for the build that follows cannot say that sentence. Not because they are dishonest — because of where their incentive sits.

An AI system will take the easiest route to looking successful. Make sure the person confirming it is successful has no reason to take the easy route too.