AI doesn’t fail loudly

In March I wrote here that AI doesn’t take jobs, leverage does, and I meant it. I still mean it. I’m a gas technician with no computer science background running a business on software I built myself, and none of that exists without AI as a collaborator. Everything I said about leverage holds.

This is the part I left out.

The assumption most people carry into this is that when AI gets something wrong, you’ll be able to tell. It’ll make up a citation, or invent a function that doesn’t exist, or say something obviously stupid, and you’ll catch it and move on. That kind of failure happens and it costs you almost nothing, because it announces itself. The failures that actually cost you are the ones that look completely fine.

Here’s the shape of it, three times over.

I had a piece of code that uploaded a file to a service. It ran clean. No error, no exception, no warning, exit code zero, and the log line at the end said the upload had completed. It had not completed. The file arrived on the other end with nothing in it. The cause was a quirk in the runtime I happened to be on, where a particular way of handing over binary data silently sent an empty body instead of the bytes. Nothing about the code was wrong to read. It was wrong to run, on that version, in that place, and the only symptom was an empty file somebody would have found later.

I had a page that looked perfect on my machine and shipped with no styling at all. The class names were being assembled from variables at runtime, and the build tool that strips out unused styles could not see them referenced anywhere, so it did what it was designed to do and removed them. Locally, in development, none of that stripping happens, so it looked right every single time I checked it. The failure only existed in the one environment I wasn’t looking at, which is the one customers use.

And I asked for a plan once and got a good one. Well reasoned, properly sequenced, clearly written, and it proposed building three tools that already existed on my machine. Not because it was careless. Because I hadn’t told it what I already had, and rather than stopping to ask, it filled the gap with something reasonable. That’s the most dangerous version, because there’s nothing to catch. The code would have worked. I’d just have ended up with two of everything and no idea why my system got harder to maintain.

None of those three announced themselves. Every one was syntactically correct, logically defensible, and confidently delivered. They were wrong only at the point where they touched reality, and reality was somewhere I hadn’t looked yet.

Which is the exact failure mode I’ve spent twenty years dealing with in basements.

A furnace can run wrong. It fires, the burners are lit, the blower comes on, the house gets warm, and the customer has no complaint. From four feet away everything about it looks like a working appliance. The combustion can still be badly out. You cannot see that. You cannot hear it. A blue flame is not proof of anything useful, and the guy who tells you he can eyeball a good burn is telling you he doesn’t measure. The entire reason we own combustion analyzers is that a system which appears to be working is not evidence that it is working, and the trade learned that the expensive way, over decades, with people dying.

So the discipline is: you don’t trust the appearance, you measure at the point that matters. You don’t accept the report, you take the reading. And when something is off, you change one thing and take the reading again, because if you change three things you learn nothing about which one mattered.

That is the whole answer to AI, and I did not have to invent any of it. I already had it. It came with the trade.

Practically, for me, it means a few things. I verify at the point of use rather than the point of output, which means the question is never “does this code look right” but “did the file actually arrive with something in it.” I test in the place the thing will actually live, not on my own machine, because “it works locally” is a statement about my laptop and nothing else. I change one variable at a time even when I’m certain, especially when I’m certain. And when I ask for a plan, I start by saying what already exists, because the model will not go looking and will not usually ask.

I want to be clear that this is not a piece arguing against AI, and I’d be a hypocrite writing one. I built a company on this. The booking, the dispatch, the invoicing, the intake pipeline that turns a voicemail into a job card without me touching it: all of that came out of a year of working with these tools, and there is no version of my life where I write that by hand. The leverage is real and it is enormous and I would tell anybody to go get some.

What I’d add now, and what I’d have wanted somebody to tell me a year ago, is that the leverage is not the skill. The tool will produce more work than you can check, faster than you can check it, and it will produce that work in a register of total confidence whether it’s right or not. The scarce thing is not generation anymore. The scarce thing is the person who knows how to verify, who knows which measurement actually settles the question, and who is temperamentally unwilling to accept a thing that looks fine as proof that it is fine.

And I think that’s genuinely good news for people in the trades, who tend to assume this whole world isn’t for them. The habit that makes you good at this is not a programming habit. It’s the habit of the guy who puts the probe in the flue when the customer is standing there telling him the furnace works great. It’s a suspicion of appearances, earned the hard way, in a field where appearances get people hurt. Nobody ever told you that was a transferable skill. It’s the most transferable one you have.

It turns out the most useful thing I brought to software was twenty years of refusing to believe a system just because it was running.