Could chat-GPT be renamed Johnny-come-lately? In the lyrics of an old Eagles song, Johnny was the new kid in town and everybody loved him.

Or do they? And who is this new kid in town anyway? Is he the wunderkid bringing us a step closer to achieving artificial general intelligence or rather “an off-ramp on the highway towards human level AI” (Yann LeCun quote)?

There are hundreds of examples on the internet of users posting absurd chat-GPT answers to questions.
Open AI, the company behind chat GPT, has recently added the following disclaimer: “while we have safeguards in place, the system may occasionally generate incorrect or misleading information and produce offensive or biased content. It is not intended to give advice”.
The definition of “occasionally” probably depends on the kind of questions posed, but examples vary from not being able to answer “what was the color of the white horse of Napoleon” to getting simple arithmetic wrong.

The grammar and spelling are usually perfect but semantics, i.e. the relationship of the text to the real world, can be significantly lacking.

In part, this should be obvious. To learn is to form a model of the world; of course “world” means whatever we experience. In a large language model, the definition of “world” is different from ours – the world of a large language model is made of words, sentences, grammatically correct constructs. The only world for which a large language model can build a model is therefore not the world we live in but the world of syntax. It is to be expected, therefore, that a large language model would be syntactically correct. However, while the written world should accurately describe the physical world, it may not be enough to allow a large language model to be semantically correct. Even were large language models to have the capability to fully learn and create a model of the world to which they are exposed, they are no more able to create a model of the real world than the group of people in the allegory “The Cave” described by Plato. They live chained to the wall of a cave all their lives, facing a blank wall and watching the shadows projected on that wall, the shadows of the objects of the real world.
The world on which chat GPT and other large language models are trained, (similarly to the shadows), is not an accurate representation of the real world.

As the saying goes, “Garbage in, garbage out”, and this is true regardless of how “intelligent” a model might, in theory, be or become. So, why does everybody love chat-GPT?
Chat-GPT is truly good at some tasks that require only lexical and syntactical abilities and this may be enough for lots of people. In addition, despite its tendency to, as Gary Marcus puts it, “hallucinate”, obviously this does not happen all the time.

What then has brought a group of leading AI researchers to request a moratorium of six months on any training of language models larger that GPT-4?
Leaving aside pseudo-conspiracy theories that suggest that people in other companies wanted the time to catch up with Open AI, which has gone from being an open source company to a closed traditional one, many people are genuinely concerned about the potential dangers.
This is not new. As early as 2016 an open letter signed by leading AI researchers stated, among other things, that “AI research is progressing steadily, and its impact on society is likely to increase”.
Now, they claim “Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources”
In short: “AI systems with human-competitive intelligence can pose profound risk to society and humanity”.

Perhaps less ominous but certainly more immediate are concerns about privacy as the recent ban by the Italian privacy regulatory agency shows.

This may appear paradoxical at first: large language models are far from intelligent. As Gary Marcus put it, they are just fancy “autocomplete on steroids”. How can we, in the same breath, criticise their supposed stupidity and warn about the profound risks to humanity?
The apparent contradiction did not escape the notice of Gary Marcus who clearly states that while he “still does not think that large language models have much to do with super intelligence or artificial general intelligence… you don’t have to be super intelligent to create serious problems”.

Part of the concern is that, while these models are not very intelligent, and may just be slaves chained inside a cave watching shadows, people are increasingly marveling at GPT models perceived intelligence.
And while chat GPT-4 may still occasionally hallucinate often enough to remind us that it does not understand the real world, what will happen with a GPT-5 or 6 or 7 or n>>4 that remains ignorant and stupid about the real world but smart enough to be able to conceal it?
The occasional hallucinations may become sporadic, then rare, then un-noticeable and we will end up with widely deployed AI, allowing large language models to access pretty much everything. But they would still be the un-intelligent models with the potential to hallucinate.

“Fooled by Randomness” is a 2001 book by Nicolas Taleb, followed by his 2007
book titled “The Black Swan” in which he describes what happens when rare events happen. Rare events that we know are rare but not impossible. In fact, they are actually unavoidable though rare. What would then happen with a “black swan” type of hallucination after we have allowed our large language model to operate increasingly complex systems?

Stanislaw Petrov was a lieutenant colonel of the Soviet Air Defense Forces.
In 1983 he was the duty officer at the command center for the Soviet nuclear early-warning system when it reported that a missile had been launched from the United States towards the Soviet Union.
The automated system reported that five more missiles were launched.
Mr. Petrov correctly reasoned it was a false alarm and did not initiate the protocol that may have initiated a retaliatory nuclear attack possibly resulting in a large scale nuclear war between the US and the Soviet Union.
Would a large language model, with no data about previous nuclear attacks, with no understanding or even knowledge of the real world, have behaved the same? While we may never entrust large nuclear arsenals to an artificial intelligence, this example clearly exemplifies the type of dangers that deployment of Al models with no understanding of the real world can produce.

Examples of black swan events literally litter human history and globalization and systems interoperability only increases their reach.
People list catastrophic events such as World War I or past global stock market crashes. But there are many examples, less catastrophic, that could be listed, such as the flash crash on May 6, 2010 that caused a rapid unexpected drop of the US stock index caused by false sell orders being placed by trading algorithms.

By creating GPT-n models, that do not understand the world but are complex and accurate enough, and relatively ubiquitous, we would create models with a built in predisposition to create more black swan events, caused by their increasingly rare but inevitable tendency to ”hallucinate”.

Continuing on the off-ramp on the highway to AGI, and continuing in that direction, will make us feel these “hallucinations” will eventually be gone forever only to come back to haunt us as a potentially destructive black swan event.