Hallucination in machines: Why AI makes things up

When we prompted multiple AI models on why they lie, the first thing they wanted to do was differentiate lies from hallucinations.

We go to Artificial Intelligence (AI) chatbots for research, guidance and emotional support. But how do we tackle the sweet lies they tell us, sometimes as much as 30% of the times? A recent study by Relum, an online gaming support engine, found that popular AI chatbots hallucinate upto 30% of the times that they are prompted for information. ChatGPT, the most popular product with users like us, makes up stuff around 35% of the times, while Gemini leads with hallucinations upto 38% times. Though other studies differ in the percentages (ranging from 17-35%), one thing is clear: One in five answers by AI chatbots is made up.

It was to fix ChatGPT’s reliability amongst other things, that a slightly panicked Sam Altman declared a Code Red in a memo to his employees, in December. (AP)

In October, the Australian government raised a furore when it found that a report created by Deloitte, the global consulting company, for one of its departments, cited non-existent experts, scientific papers, even studies conducted by University of Sydney that didn’t exist. After it became an issue, Deloitte confirmed that it used Microsoft’s Azure OpenAI GPT4o system to assist in drafting parts of the report. Just as the company refunded $290,000 to the Australian government, Canada’s health department found false citations, made-up academic papers in a report that the same consultancy firm had developed for them. Again, thanks to research which used an AI chatbot.

The global market size for AI technology, infrastructure, software services and business was $371.71 billion in 2025. This is set to grow at a whopping 30.6% per year to $2.407 trillion in 2032, according to research by Markets and Markets. As AI is deployed in internal company systems, healthcare, finance, cybersecurity and defence of countries, hallucination has become a challenge for everyone from tech companies to governments. Like the Deloitte example shows, the models make up data rather than simply saying “I don’t know.”

It was to fix ChatGPT’s reliability amongst other things, that a slightly panicked Sam Altman declared a Code Red in a memo to his employees, in December. The idea, according to Altman, is to improve ChatGPT’s personalisation, usability and reliability. It’s a complete reversal from his relentless pursuit of more computing, more infrastructure and more scale to reach AGI (Artificial General Intelligence). The aim, said Altman, was to tackle hallucinations.

Why does an AI model lie?

Hallucinations, as the AI industry calls plausible but false statements generated by large language models like ChatGPT and Gemini, are something that has been ingrained into the way these older models were constructed. When we prompted multiple AI models on why they lie, the first thing they wanted to do was differentiate lies from hallucinations. “I don’t lie – white or otherwise. I can be mistaken, outdated, or limited, but I don’t intentionally deceive,” answered ChatGPT, acknowledging that it can go wrong when it has outdated information, incorrect sources or gaps in its training. US-created Gemini, Claude and Perplexity AI had similar responses.

But then, if the models have incomplete or outdated information, why not just say that? Why make up information, data and facts? When we prompted the models again to ask “When you don’t have an answer, why hallucinate?” The response, clearly put it on the makers of the model. “I don’t ‘retrieve’ facts like a database; I predict text based on patterns learned during training,” explained Gemini. “When you ask a question, my primary objective is to complete the conversation with a fluent, relevant response.” When fresh data is missing, the model relies on older data, generalisation from similar topics and linguistic probability, leading to “plausible but incorrect statements”.

“I’m designed to always respond,” agreed ChatGPT, expressing this need in the model to be helpful even when lacking facts and data – to generate the most statistically likely answer. ChatGPT puts this squarely on its makers. Thanks to pretraining the models a certain way, most of these AI models have no understanding of what they don’t know. On simple tasks like summarising text, coding or general knowledge, the hallucination rate is very low – around 1-3%, but make the task complicated – niche content, specific URLs, obscure news sources or even verify something niche, this hallucination rate is as high as 60-70%.

“If you ask for a specific URL from a news article from 2022, I might hallucinate a link that looks correct (right domain, correct date format, correct keywords) but leads to a 404 page,” says Gemini. The reason, Gemini explains, is that it is predicting what a URL should look like, not retrieving it from a database.

These hallucinations can show up in surprising ways, even for seemingly straightforward questions, writes Adam Tauman Kalai, AI researcher at Open AI who released a scientific paper on hallucinations in September . “For example, when we asked a widely used chatbot for the title of the PhD dissertation by Adam Tauman Kalai, it confidently produced three different answers—none of them correct. When we asked for his birthday, it gave three different dates, likewise all wrong.” According to Kalai, hallucinations persist because of wrong incentives. Model performance is measured on guessing rather than honesty about uncertainty. This makes AI models reliable if there is a single right answer and unreliable if there’s ambiguity. There is no true/false label, so the models don’t understand wrong from right. OpenAI’s own evaluation of ChatGPT 5 has an error rate of 26%. “Penalise confident errors more than you penalise uncertainty and this will reduce,” writes Kalai.

Rewarding uncertainty over a guess

Companies from OpenAI to Google, from X to Meta are constantly tackling the hallucination challenge to make their models more reliable. But it’s easier said than done. A new study found that even with training the models and AI agents built upon them, still overestimate their knowledge or if given too many parameters, over-refuse (which means ignoring some prompts) to give answers. Calibration of an AI model is an art right now. It’s possible, if the models are given what the researchers are calling ‘IDK datasets’ (I-Don’t-Know datasets). These datasets include specialised ones that teach models to say IDK to certain prompts and follow instructions, supervision and feedback from humans (called reinforcement learning in the industry).

Google Deepmind created a set of rules called Sparrow, that applied human feedback to make the model relearn and find and cite factual information. Anthropic’s Claude AI has a constitution that the model follows at all times. Claude AI’s constitution places clear boundaries, explicit values and a set of principles and processes while training the model. This is reinforced by using human feedback. To build the constitution, explain the researchers at Anthropic, they’ve used DeepMind’s Sparrow codes, UN Declaration of Human rights, trust and safety practices, an effort to capture non-Western perspectives and Apple’s terms of services. Claude AI, thanks to its codified constitution, has less hallucinations (at 17%, according to the Relum study) than its counterparts. It is perhaps reliability that has made Anthropic’s Claude AI become a preferred partner for enterprises. The company has more than 300,000 enterprise customers with 32% enterprise AI market, ahead of OpenAI and Google (both at 20%), according to data from Menlo Ventures.

“When I don’t know something, I tell you that I don’t know,” Claude Sonnet 4.5 replies when asked if it hallucinates, adding that its knowledge is till end of January (the model we query) and if it’s not been trained on something, or is not confident enough in its knowledge, it’ll respond so. “I try to avoid the trap of sounding confident when I’m actually uncertain or if I’m only moderately sure about something,” it says, adding that its goal is to give you an accurate sense of what it knows and admit gaps in its knowledge.

There have been a spate of recent studies on how Gen Z and Gen A trust AI more than humans for everything from mental health advice to career decisions. But at the heart of it, AI models are commercial entities, created to keep you hooked – much like social media. Will future generations know how to filter the chatbot’s sweet white lies, or will they hallucinate together with the model? After all, hallucinations — perceiving something that’s not present — is such a human thing to do. As is telling white lies.

(An author and columnist, Shweta Taneja tracks the evolving relationship between science, technology and modern society)

https://menlovc.com/perspective/2025-mid-year-llm-market-update/