Posted on
Somewhere beyond the reach of Procurement, in a region of spacetime that no enterprise architecture framework has successfully mapped, there exists a parallel universe in which demos take place.
Scholars have long suspected its existence. The evidence is everywhere.
In this universe, the data is clean. Not "we ran a script over it last quarter" clean... properly clean, the way no data has ever been in the entire history of recorded commerce. The networks are stable. The stakeholders are reasonable, aligned, and present, which alone should have tipped everyone off that something was deeply wrong.
The model, in this universe, performs flawlessly. It has been shown the questions in advance. This is not cheating, not exactly... It is more that the demo has been arranged, with great care, so that the model is never once asked anything it might get wrong.
Some vendors live there. It is a lovely place.
They give tours, and the tours are excellent: guided, narrated, and conducted entirely within a sealed environment where nothing unexpected has ever been permitted to happen.
Strangely, at no point does anyone open the cupboard marked "Edge Cases", because the vendor has thoughtfully welded it shut.
You, meanwhile, live in this universe.
The one with the gravity.
In this universe, the data is held together by a spreadsheet on Janet's old laptop. Remember the one from a previous post on dodgy spreadsheets? Its still out there.
In this universe, the network goes down every second Tuesday for reasons that three separate teams have agreed are someone else's problem. And Legal has just wandered in, pale and holding a printout, to ask what exactly the model has been doing to the customer records since March.
The model, it turns out, has been improvising. It was confidently wrong about a great many things, at scale, in writing, to customers. It was never told not to be, and, more to the point, nobody had checked whether it could tell the difference.
This is the part the demo skips. Especially with AI these days... yes, we had to talk about AI here.
The numbers bear it out, and have for some time. RAND found that more than 80% of AI projects fail. That is twice the failure rate of ordinary IT, which was hardly setting records to begin with. MIT's NANDA Initiative put the failure rate of generative AI pilots at 95%.
And S&P Global reported that in 2025, 42% of organisations abandoned most of their AI initiatives, up from 17% the year before. That is the sort of trend line that, in any other context, would prompt a meeting.
Here is the genuinely instructive part, and it's instructive in the way that a banana skin is instructive right up until it is your banana skin.
RAND found that the failures came from both ends at once. At one end, leaders who understood "AI" purely as a word, a good word, a word that tested well in board papers, without quite registering that it referred to a thing that did stuff, much of which required data they did not have and infrastructure they had not built. At the other end, the technology itself: models that were brittle where the demo was smooth, that fell over on messy real-world inputs, that hallucinated confidently, and that needed far more careful engineering than the polished pitch had ever let on.
So, no... the technology is not simply "fine." It is genuinely hard, often immature, and quite capable of failing on its own merits.
But that is rarely the part that kills a project. What kills a project is the gap between the two universes: the institutional, organisational, Tuesday-shaped gravity that the demo has been carefully engineered to exclude. The technology's real limitations and the organisation's real mess arrive together, at month nine, holding hands.
A demo is a magic trick performed in a sealed room, by a magician, using a deck the magician brought.
Production is the same trick performed during an earthquake, on a borrowed deck with three cards missing, by a committee, on a budget, while a man from Legal stands at the back holding a printout and clearing his throat.
So when a vendor's pitch leans on the demo, when the deck glows and the synthetic data behaves and everyone in the room begins, dangerously, to nod, ask them one question.
Ask what month nine looked like on their last three production rollouts. Not month one. Month nine, when the novelty has worn off, the edge cases have escaped the welded cupboard, and Janet has handed in her notice.
Then time the silence.
The silence is the product.
