A kid circles the right answer. Our AI marks it wrong. And every test still passes.
I used to think the scariest AI mistake was getting the answer wrong. I was wrong. The scary one is getting it confidently wrong, while the whole team stares right at it and nobody notices.
This is a story about a bug we hit recently. Not much code in it, but the product lesson is one I want to write down.
The demo looked great. Then what?
We're building an AI reading platform for upper-elementary kids. One feature works like this: a student circles their answers on a paper worksheet, snaps a photo, and the AI reads every question, grades it, and gives feedback.
The demo looked great. Every question read perfectly. Everyone nodded.
Then I actually dug into the data, and my stomach dropped.
One question type was marking 91% of student answers wrong. Out of 51 lessons, 50 were affected.
And every automated test was green. No error message. Nobody had caught it.
The AI wasn't dumb. We had two answer keys.
How does that happen?
The worksheet printed for students shows a letter-to-word table: A is one word, B is another, C is another. A student circles A, meaning the first word.
But in our code, we weren't checking against that table. We were checking against a different list of words, sorted in a different order. The order in that list didn't match the order printed on the worksheet.
So a kid circles A (the right answer), the system grabs a different word to compare against, and a child who got it right gets marked wrong.
I pulled the first five letters: A through E, every single one wrong.
The problem wasn't the AI. We had two "answer keys," and we'd never said which one counted.
Honestly, if we couldn't tell which one was right, how could we expect the AI to guess?
The part that actually chilled me
The bug itself is easy to fix. The part that chilled me was something else.
The decision about "which one is the source of truth" had actually come up in a meeting. But it only lived in the meeting notes, in someone's head, in documents scattered across the place. Nowhere did anyone write down, plainly: "this field is the one true answer key."
I counted. The spec for how this one feature is supposed to work was spread across seven places: product docs, meeting notes, code comments, task lists, chat messages.
So when the AI goes to write this piece of code, it has to read all seven places and guess which one matters, or it misses the one that did.
Here's the counterintuitive bit worth remembering: we assume that the more context you feed an AI, the more accurate it gets. The opposite is true. When the "truth" itself is scattered and contradictory, the more you feed it, the more confidently it picks the wrong one.
What I did: force the fuzziness into the open
So I did one thing. I made the spec modular. In plain terms, three moves:
First, each feature gets one plain-language "truth" document that says which field counts, what can change, and what can't.
Second, that same truth gets written as a test that runs itself, so the computer checks whether the code actually matches the spec. If they don't line up, it goes red. No relying on someone to remember.
Third, before the AI touches this code, it reads that one small doc, instead of swallowing the entire product spec.
I used the 91% bug as the first guinea pig. I wrote the first spec-as-test, ran it, and it immediately quantified the whole thing: 51 lessons, 91% wrong.
We'd assumed it was "a few lessons acting weird." Turned out it was systemic. The whole batch.
The real lesson has nothing to do with code
The biggest takeaway wasn't technical. It was this:
The biggest risk in an AI product is never that the AI makes mistakes. It's that your team never made "what's correct" explicit, and then expected the AI to guess right anyway.
And it has a shelf life. Six months from now the team will have turned over, and if "which one is the truth" only lives in one person's head, the new people, and the AI, will step on the exact same landmine.
So what we're really doing isn't "governing the AI." It's forcing ourselves to write down the decisions we once waved past.
The AI is more like a mirror. What it reflects is the part we never bothered to think through.
Open your own product docs right now. Can you find the one source of truth? Or does it only live in someone's head?
Next time an AI hands you a confident answer, maybe the first question isn't "is the AI right?" It's "did we ourselves ever make it clear?"