There have been many reports of Artificial Intelligence (AI) programs, based on Large Language Models – LLM, providing incorrect information in the form of confident statements that are not true. For instance, in May 2023, Chat GPT produced six false case precedents for a lawyer, Stephen Schwartz in The Southern District of New York. When the nonexistence of these cases was discovered after they had been submitted to the Court, ChatGPT continued to assert that the precedents were authentic.
These “plausible sounding’ random falsehoods” (Wikipedia) were initially called ‘hallucinations’ based on the concept of human hallucinations. The ideas of parallels between human hallucinations and this AI phenomenon has been criticised as anthropomorphism – a problem already identified when human qualities are glibly ascribed to animals – which can interfere with truly identifying and understanding animal behaviour. And, as Charles Seife ( OpenAI’s bot wrote my obituary It was filled with bizarre lies. slate.com) many AI programs have been developed on a paradigm of simply simulating humans, not actually functioning as humans. Others have made similar comments.
Another criticism of the term ‘hallucinations’ here is that the word refers to a form of false perception – of vision, hearing, taste etc. The AI hallucination doesn’t involve any perception at all. So, some authors are now referring to this phenomenon as ‘confabulation’, a term which sits better with me.
It may be useful and, if not useful at least interesting, to consider and compare this AI phenomenon of confabulation with some similar human experiences.
Confabulation by humans.
The common clinical meaning of the word is: “to fabricate imaginary experiences as compensation for loss of memory”. People with Alzheimer dementia, Korsakoff syndrome and other brain problems often confabulate, speaking convincingly and confidently of events which have not happened. In this definition the phenomenon is put down exclusively to memory loss but, since that definition was made it has become clear that another brain malfunction apart from memory problems is usually involved.
The person confabulating, as well as having memory loss, has usually lost the ability to tell the difference between real memories of real events and false memories arising from their internal thoughts and imagination or sometimes arising from some external experience. For instance a person experiencing dementia might view a television travel show about Venice and then believe they have visited Venice when they haven’t.
We see people who have very poor memory but no confabulations. And we see, rarely, people with brain problems who have confabulations although their memory functions are intact.
The second, non-memory part of the confabulation problem happens when there is damage to a particular area of the brain called the Ventromedial Prefrontal Cortex – VMPFC, especially if this damage is on the right. It appears that part of the function of this part of the brain involves monitoring the source of information being considered – whether from within the person’s brain or from experience outside the person.
Another related function of this part of the brain has been called the ‘doubt tag’. Normally if our brain picks up some discrepancy in information that we are considering, we experience doubt, and that may cause us to consciously review that information for its accuracy and validity. Confabulating people show an abnormal level of certainty about their false memories. They appear to lack this doubt tag.
Which brings me to the question of whether people and machines who confabulate are lying. The word ‘lying’ suggests that a person is aware of the truth but deliberately chooses not to stick to the truth. That is, they have an intent to deceive. Confabulating people with memory problems are, by this definition, not lying. Chat GPT, dealing with statistical probability of words, and to a very limited extent, if at all, the concepts behind the words, is surely not ‘aware’ of the truth. I understand however, that AI has surprised many people because it appears to generate something close to awareness of concepts rather than just awareness of words.
Confabulating people whose memory is intact but who have a failed fact-checking and doubt tag system might just qualify as lying but certainly not in the usual sense of the word. One person with VMPFC damage I saw as a patient did seem able to suppress communicating their confabulated narratives under social pressure. I don’t know whether they still constructed these narratives but kept them to themselves. Perhaps this person’s condition would be better labelled as ‘pathological lying’ rather than confabulation. The fascinating condition of ‘pathological lying’ needs a whole blog on its own. I will skip over it here.
AI confabulations: The mechanisms behind AI confabulations are different from human brain processes. The AI used in programs like ChatGPT uses Large Language Models – LLMs. Some of these, such as training program GPT3, analyses vast amounts of human produced written material to develop predictions of what words and phrases are likely to follow a word or phrase being examined. This has been described in a cynical oversimplified way as a magnified version of the autocomplete available for written phone messages. So it generates each next word based on a sequence of previous words, including the words it has itself generated. This can lead to a cascade of self-reinforcing confabulations.
There is much more to LLM than this brief description. This includes ‘back propagation’ where the program learns from its mistakes. This is, in some way similar to the human ‘doubt tag’.
Later versions of GPT have developed improved accuracy and have also improved the program’s ability to acknowledge ignorance – to refuse to answer questions when the program doesn’t know the answer. Some programs also use a function called RLHF – Reinforced Learning from Human Function, allowing input from human sources.
Another interesting concept in AI is that of ‘temperature’. LLMs operate on the basis of probability, and temperature settings can determine the range of probability values under which the program can operate. For example the program might look at the probability of the next word in the sentence “The cat is …” and find the following: ‘playing’ 0.5; ‘eating’ 0.15; ‘flying’ 0.05. That is, ‘flying’ is very unlikely while ‘playing’ is quite likely. If the program is assigned a low temperature, accepting more high probability words, then it is likely to choose ‘playing’ over ‘flying’. If it is set at a more imaginative, creative high temperature the choice is more likely to be ‘flying’.
My anthropomorphic take on this ‘temperature’ aspect of AI programs notes that a small proportion of people with Frontotemporal Dementia – FTD, particularly when early degeneration is mostly in the Temporal lobe(s) have been observed to develop greater creativity than they previously displayed. The tragic progression of the FTD ultimately overwhelms this creativity. And, it must be said that these creative behaviours are often odd, driven, and repetitive. But there are similarities and parallels perhaps in this creative freedom of expression to the confabulations of dementia and the ‘creative’ outputs of ChatGPT or of DALL-E 2 graphic images. Does creativity require less probable, expected, predictable associations?
I should point out that my book ‘Your Brain etc.” contains quite a lot of discussion about confabulation (p.98 – 105)) and pathological lying (105, 211-216, 378) (uninvited plug for book).
How a symptom of dementia informs our understanding of creativity (statnews.com)
OpenAI’s bot wrote my obituary. It was filled with bizarre lies. (slate.com)