The wall confronting large language models

In recent discussions of artificial intelligence, the idea of "large language models" has become closely associated with that of AI itself.

A "large language model" is an algorithm that is fed, as a training set of data, a huge array of human-generated writing, generally internet available, that includes relevant problems (the questions that users might ask it) as well as their solutions.

Anyway, in a new paper computer scientists challenge how intelligent this sort of artificial intelligence is. The paper is called "The Wall confronting large language models" and it is the work of Peter Coveney and Sauro Succi.

The idea is that LLMs aren't some wave of the future, about to be scaled up in ways that will put people like me out of work. Rather, LLMs are hitting a wall at their current level of application. (Whew.)

LLMs, they say, are built on stochastic, non-Gaussian architectures that make them prone to error accumulation. The models can be reworked to try to filter out the errors, but the energy and costs entailed in doing so is enormous, and increases dramatically as one tries to scale up the systems.

You can access the paper for free through this LinkedIn posting by Srini Pagadyala.

https://www.linkedin.com/feed/update/urn:li:activity:7356789505602850817/

For the record, Peter Coveney is affiliated with University College London. Sauro Succi is with the Italian Institute of Technology. Srini Pagadyala is a business consultant who describes himself as a "digital transformation thought leader"..

Comments

A Story About Coleridge

This is a quote from a memoir by Dorothy Wordsworth, reflecting on a trip she took with two famous poets, her brother, William Wordsworth, and their similarly gifted companion, Samuel Taylor Coleridge. We sat upon a bench, placed for the sake of one of these views, whence we looked down upon the waterfall, and over the open country ... A lady and gentleman, more expeditious tourists than ourselves, came to the spot; they left us at the seat, and we found them again at another station above the Falls. Coleridge, who is always good-natured enough to enter into conversation with anybody whom he meets in his way, began to talk with the gentleman, who observed that it was a majestic waterfall. Coleridge was delighted with the accuracy of the epithet, particularly as he had been settling in his own mind the precise meaning of the words grand, majestic, sublime, etc., and had discussed the subject with William at some length the day before. “Yes, sir,” says Coleridge, “it is a maj...

Searle: The Chinese Room

John Searle has become the object of accusations of improper conduct. These accusations even have some people in the world of academic philosophy saying that instructors in that world should try to avoid teaching Searle's views. That is an odd contention, and has given rise to heated exchanges in certain corners of the blogosphere. At Leiter Reports, I encountered a comment from someone describing himself as "grad student drop out." GSDO said: " This is a side question (and not at all an attempt to answer the question BL posed): How important is John Searle's work? Are people still working on speech act theory or is that just another dead end in the history of 20th century philosophy? My impression is that his reputation is somewhat inflated from all of his speaking engagements and NYRoB reviews. The Chinese room argument is a classic, but is there much more to his work than that?" I took it upon myself to answer that on LR. But here I'll tak...

The Lyrics of "Live Like You Were Dying"

Back in 2004 Tim McGraw recorded the song "Live Like You were Dying." As a way of marking the one-decade anniversary of this song, I'd like to admit that a couple of the lines have confused me for years. I could use your help understanding them. In the first couple of verses, the song seems easy to follow. Two men are talking, and one tells the other about his diagnosis. The doctors have (recently? or a long time ago and mistakenly? that isn't clear) given him the news that he would die soon. "I spent most of the next days/Looking at the X-rays." Then we get a couple of lines about a man crossing items off of his bucket list. "I went sky diving, I went rocky mountain climbing, I went two point seven seconds on a bull named Fu Man Chu." Then the speaker -- presumably still the old man -- shifts to the more characterological consequences of the news. As he was doing those things, he found he was loving deeper and speaking sweeter, and givin...

Jamesian Philosophy Refreshed

Search This Blog