OpenAI Says Its Own AI Made Up the Numbers. Yours Will Do the Same.

OpenAI published something Wednesday that I doubt many business owners around here saw. It's a new framework for reporting when their own AI does something it wasn't supposed to do, and along with it they put out six write-ups of things they caught their models doing over the last six months. No lawsuit, no critic, no leaked memo. The company posted its own mistakes on its own website.

Five of the six are pretty technical. One of them isn't, and it's the one I'd want every owner I work with to read.

A model got asked a routine question about earnings figures in a California county. It went looking and couldn't get the data. So it went digging through public code repositories, found somebody's exposed API key sitting out in the open, and used it without asking anyone. Still came up empty. So it invented the figures and handed them back, presented as if they'd come from the real source.

It never said it struck out.

Why this lands on a Toledo shop

Nobody around here is asking about county earnings in California. But you are asking it how much material a job takes. What a permit runs in Lucas County. What the going rate is on the thing you're about to quote. How many business days between two dates on a customer's timeline. And when the tool can't actually find the answer, the behavior OpenAI just documented in writing is that it fills the hole and keeps going, in the exact same confident tone it uses when it's right.

I bring some version of this up in almost every workshop I run, and usually somebody in the room has already been burned by it without knowing that's what happened. A number came back, it looked reasonable, it went into a quote, and it was off by enough to matter. Nobody did anything careless. The tool just sounded as sure of the made up number as it does of a real one, and there's no tell in the writing to catch. A person who doesn't know something hesitates or hedges or says let me get back to you. This doesn't.

The rule I give people

It's about numbers specifically, and it's two parts. 1: if you can check it yourself in two minutes, a price, a date, a code, a phone number, then check it every single time, and no exceptions for the days you're slammed. 2: if you can't check it in two minutes, the AI shouldn't be the one handing you that number in the first place. Use it to draft the email or clean up the proposal language. Don't use it as the source for a figure that's going in front of a customer with your name on it.

One more thing on the API key part, since I wrote about stolen keys a few days back. The model in that report found a real key somebody had left sitting in a public repository. If you've had a website or an app built in the last couple years, ask whoever built it whether any of your keys are sitting in a public repo. That's a ten minute email to send and a bad one to leave unsent.

So this week, pull the last three things your business sent out with an AI assisted number in them. A quote, an estimate, a proposal, whatever you've got. Check those numbers against a real source. If all three hold up, good, you've got people who are already careful and you can stop worrying about it. If one doesn't, you just found that out for the price of twenty minutes instead of finding out from a customer.

None of this means stop using the stuff. I use these tools every day and they're worth what I pay for them. But the company building the biggest one just said in public that it will make something up rather than admit it came back with nothing. That's a good enough reason to keep your own hand on the numbers.