How to tell whether your software supplier is doing a good job

There is plenty of advice on choosing a software supplier. There is almost none on the harder situation: you are already six months in, something feels wrong, and you have nobody in-house who can tell you whether it actually is.

You cannot read the code. The updates sound reasonable. The estimates keep moving, but there is always an explanation. And the one thing you are certain of is that asking the supplier whether the supplier is doing a good job will not settle it.

Here is what to look at instead. None of it requires you to be technical.

Start with the estimates, not the code

The most reliable signal is not quality — it is how estimates behave over time.

Healthy estimates get more accurate as a project proceeds, because uncertainty is being removed. If you are six months in and the remaining work is still described in the same round numbers as at kick-off, nothing is being learned. That is the tell.

Two specific questions:

• When an estimate moves, are you told what was discovered? "It's taking longer" is not an answer. "The client data has three formats we hadn't seen, so extraction needs reworking" is. The second means somebody is learning; the first means somebody is guessing.
• Were the original estimates given before or after discovery? Roughly half of all rework in software projects traces back to poorly gathered requirements. An estimate produced before anyone understood the problem was optimism, and you are now paying for it.

Ask when you last saw it working

Not a demo. Not a screenshot. The actual thing, running, that you could click.

You should be seeing working software incrementally — every two or three weeks at worst. Iterative delivery is not a process preference, it is a risk control: it creates regular chances to change course while changing course is still cheap.

If everything is coming at the end, you have no visibility and no leverage. By the time you see it, the money is spent and your only options are accept or litigate.

A softer version of the same problem: you see demos, but always driven by the same person, always following the same path. Ask to click it yourself. The difference in what you learn is remarkable.

Three questions that reveal more than they should

"What's your test coverage on the critical paths?"

You do not need to understand the answer to learn from it. Below 40% on critical paths is a yellow flag. Zero automated tests is a red one. But the real signal is whether they know the number. A team that has never measured it is telling you something.

"What's the riskiest part of this build, and what are you doing about it?"

Every real project has one. A supplier who says "nothing really, it's all fairly standard" is either not thinking or not telling you. The answer you want names something specific and describes a mitigation.

"If you were hit by a bus tomorrow, what would it take for someone else to pick this up?"

This tests documentation, structure and lock-in in a single question, and it is very hard to bluff. Listen for whether the answer involves handing over a repository and a document, or whether it involves someone spending weeks in their head.

Watch for the vocabulary shift

A specific pattern worth naming, because it is easy to miss while it is happening.

Early in a project, suppliers talk about your business — your customers, your process, the outcome you wanted. As things go wrong, the language shifts to their technology. Suddenly every conversation is about the framework, the migration, the refactor, the architecture.

That shift is often the first visible symptom of a project that has stopped being about the outcome. It rarely announces itself any other way.

What good actually looks like

For balance, the signals that things are fine:

• Bad news arrives early and unprompted, rather than being discovered by you
• The supplier occasionally says "that will take longer than it is worth — do you actually need it?"
• Documentation exists and is current, without you having to ask
• The person who does the work is in the room, not only the account manager
• They are comfortable with you talking directly to their engineers

That last one is worth its own paragraph. Suppliers who route everything through an account manager are managing a relationship. Suppliers who let you talk to the person building it are managing a project.

The structural problem

Here is the awkward bit. Every question above is one you have to ask the supplier, and grade using the supplier's own answers.

That is workable if you have someone in-house who knows what a good answer sounds like. Many organisations do not — which is precisely why they hired an external supplier in the first place. It is a genuine gap, and the people who feel it most are the ones least equipped to close it.

The usual fix is a design authority: someone independent who sits on your side of the table, reviews the architecture and the estimates, and can say plainly whether what you are being told is reasonable. In UK enterprise and government it is an established role with its own procurement category, though it is normally priced for organisations considerably larger than the ones who need it most.

It does not need to be expensive. It is a day or two a month, and the cost is trivial against a build budget. What you are buying is somebody with no stake in the invoice — which, as with a proof of concept, is the only thing that makes their answer worth having.