- 11:00
Is the fight against AI hallucinations destined to fail? OpenAI gives a mathematical explanation to the phenomenon, along with a radical fix that could, paradoxically, threaten the very existence of chatbots.
Like other generative models, ChatGPT revolutionized our relationship with information while continuing to deliver confident factual errors. OpenAI, the company behind the famous chatbot, published a new study that doesn’t dispel doubts but instead reveals how deeply the problem is rooted in the system’s design. These so-called hallucinations -invented, false, or biased answers- are not mere bugs; they stem from the very way language models generate text.
The OpenAI team provides a mathematical demonstration that these errors are not simply caused by flawed data. Even with a theoretically perfect training set, the process of predicting each next word based on probabilities inevitably leads to cumulative inaccuracies. The longer a chatbot’s answer, the higher the chance of error. The flaw isn’t marginal, it’s structural. When asked about information that’s rare in training data (like a little-known birth date), the error rate spikes. In one example, when researchers asked the model for one of their own birthdays, it gave a different, always wrong, answer each time. By guessing over and over, AI ends up asserting without basis.
The issue is worsened by current evaluation benchmarks. Industry rankings tend to reward AIs that answer, even at random, rather than those that admit uncertainty or refuse to respond. In nine out of ten benchmark datasets, hesitation is penalized as harshly as a blatant error. As a result, the optimal strategy for a chatbot is to always answer, even if that means hallucinating. This mechanism perpetuates, and even worsens, the phenomenon.
Market vs. Truth
According to OpenAI, breaking this cycle would require models to estimate their own confidence before replying. The idea: only allow answers when the probability of correctness exceeds a threshold, say 75%. That would reduce false statements, but at the cost of a chatbot that more often admits uncertainty. In simulations, a “cautious” system could refuse to answer nearly 30% of user questions.
And therein lies the dilemma. Users expect AIs to respond to everything, even when they don’t know. A chatbot that often says “I’m not sure” risks frustrating users and pushing them toward more assertive (and possibly less accurate) competitors. As one of the study’s authors recalled from a previous project on air quality monitoring, users preferred false but clear numbers over nuanced or uncertain ones.
The equation gets even more complex when economics come into play. Making an AI aware of its own uncertainty demands massive additional computation: generating, comparing, and scoring multiple responses before deciding whether to answer. At millions of daily queries, the energy and cost become unsustainable, except in high-stakes sectors like healthcare, finance, or defense. For everyday users accustomed to free and instant tools, such a model simply is not viable today.
In short, the current ecosystem of consumer AI actively drives models to hallucinate. Everything feeds the bias: impatient users, reward structures that favor confidence, and economic pressures that penalize caution. Unless these incentives change, “guessing in order to please” will remain the fuel that powers popular chatbots.
OpenAI’s report exposes a fundamental contradiction: for a chatbot to be both reliable and performant, society would have to rethink its expectations and the economic and technical rules that sustain AI.
Until then, the question remains: Is ChatGPT doomed to hallucinate in order to survive?
Comments