Why the AI gave your pub to someone else
A real measurement of two AI systems answering the same customer questions about the same pub found two opposite failures: one barely knew it existed, and handed its history to a rival.
We ran a real test on a pub that shares its name with a bigger, unrelated venue nearby. Twenty-five questions a real customer would actually type: where is it, who runs it, what’s the oldest pub around here. Fed to two AI systems, fifty answers in total, every one read and scored by hand.
One of the systems reads the live web before it answers. The other mostly answers from what it already knows, trained some months back and rarely rechecked against anything current. That one difference produced two completely different pictures of the same pub.
The live-search system found the right pub in 24 of 25 answers. The memory-based system found it in 8 of 25. Worse than a low count: in one answer it told the customer the pub belonged to a completely different company, a well-known rival group that has never owned it. Asked five times which pub nearby was the oldest, a title this pub can genuinely back with the historic record, the memory-based system gave the title to a rival every single time.
Two machines, two failures, one fix each
The live-search system’s mistakes were mistakes of the moment: it read a page, misread a detail, moved on. Fixable within days, because the next time it reads the web it reads the correction.
The memory-based system’s mistakes were older and harder to shift. It had learned the wrong story from thin, secondhand material, the same way a rumour keeps circulating after the truth is available if nobody’s said the truth loudly enough, in enough places, for long enough. Correcting it means changing what’s written about the pub out in the world, not the pub’s own page. It also means waiting, because that model doesn’t reread the web every day.
The root cause was almost nothing
Nowhere that a machine reads did it say, in plain text, who runs the pub and which of two similarly named venues actually holds the historic claim. Not on the pub’s own site, not on the review platforms, not on the one public reference page that exists for it. The fact was true. It just wasn’t written down anywhere an AI system was likely to find it.
That’s a cheap problem to fix and a slow one to feel the benefit of. Fix the live-search answer this month. Expect the memory-based answer to take longer, because it’s reading last year’s internet, not this morning’s.
What leaders should do next
- Ask an AI system who owns your pub and which of you holds any claim you compete on. If it gets either wrong, you have a findable problem, not a mysterious one.
- Write the disambiguating fact in plain text on your own page: the real owner, the real claim, the thing that tells you apart from anyone with a similar name.
- Get that same fact repeated on at least one page you don’t control, a review platform or a reference source, because that’s what the slower AI systems actually learn from.
- Re-test in three months, not next week. The live-search answer will move fast. The memory-based one moves on its own clock.