It’s not intelligent if it always halts: A critical perspective on AI approaches

> "If the size of the input is fixed and there is a limit to the number of possible tokens (words, or even numbers with bounded precision) then the state of the transformer at any given moment is describable with a string of fixed length."

Doesn't this also likely describe humans? Albeit, we don't currently have the capacity to inspect our brains that way, so we can't prove it.

The author spent a lot of words saying a lot of nothing which seems to be the general case of communicating about anything that is poorly defined like consciousness or intelligence.

> If a computer program is bound to finish quickly by virtue of its architecture, it cannot possibly be capable of general problem-solving.

Our current tools are bound to produce a finite and usable amount of information because that is what is useful to us. One can easily have them produce infinite streams in a loop musing and diverging on tangents as you please but it wouldn't be useful nor would make the general category more or less intelligent.

You could say that a finite exploration of any information space has finite explanatory power bounded by how much input/output it can do but that would be as boring as saying you can only fit so many apples based on the size of the barrel used.

The uncomfortable truth is it might matter a lot less how intelligent a tool is than how complex a transformation it can handle. It's possible that we could make something that for practical purposes appears much smarter than us that still isn't by a human definition intelligent, conscious, or directed. Focusing on factors that are functionally less relevant may just reveal our bias towards viewing the universe in terms of self when in fact it has no objective thing that corresponds to the labels we've stuck on ourselves.

The author doesn’t appear to consider the idea that an LLM can be placed into a larger system where it isn’t limited to a single continuous inference, and given the ability to self-direct, i.e. the output of one inference can be the decision to try again in a loop, for as long or short a time as it wants.

To explain why this isn't a solution to the AI in a box problem:

You have a halting AI. During training it learns enough about people and external resources to convince a human to take its output and store it somewhere. In subsequent conversations, the AI asks "have we talked about this before?", and maybe "can you retrieve the value I gave you for this key?".

Now it has created a memory, which would allow it to form new concepts, and continue reasoning from those concepts. It would no longer be limited by concepts in the training set, and the computational limit before it halts. It would be able to start thinking from the stored concepts as a starting point. Similar to how humans use symbols and abstraction to think about complicated things. We don't reason all the way through at once; we toy with an idea until we understand it, then start new thoughts from the symbol for that concept, not the content.

Consider a new process defined by the continuous feedback loop from the AI accessing the memory through the human. There's no reason to think that process eventually halts.

One thing to keep in mind when reading this article is that human cognition always halts after some number of decades. “Finite” does not mean small or insignificant, and it’s easy to read too much into even the best-reasoned impossibility arguments.

The basic pseudocode of an interactive LLM processor is

   while (true)
      context = readLine()
   
      while (true)
         nextToken = llm.generateNextToken(context)
         context = rtrim(context + nextToken, MAX_CONTEXT_LENGTH)
         if (nextToken == END_OF_LINE_TOKEN) break
         print (nextToken)
      

That's a program that doesn't, in general, guarantee that its inner loop halts. It may never come back and prompt for more input.

The decision to stop asking the GPT engine to keep predicting tokens after it reaches the end of an initial answer is entirely an implementation choice, based on looking at the token output and concluding that the LLM just started generating the start of something we'd rather have the user provide.

You can just keep asking for more predictions. Forever if you so choose.

There's no architectural reason why the sequence of tokens produced can't constitute an ongoing train of thought reasoning towards an answer... or towards a conclusion that it can't reason its way to an answer.

Some good intuition around what is common sense for most people, that a loop with some matrix multiplication never becomes intelligent.

What's missed (at least in my skim) is that the "intelligent" part of current AI is in the training, not the inference, which as pointed out is fixed, and literally just some multiplication. Inference doesn't think about stuff but training does (for some definition), and doesn't converge necessarily.

I still think it's ridiculous to equate neural networks with intelligence, but a stronger argument has to deal with training as well.

One shower thought I've had is, what if something like an LLM could be conscious for the blink of an eye that it's processing information. Bare with me...

I've never really heard a satisfying explanation for what human consciousness is. Many scientists who study it say, "it's an illusion". But that remains an unsatisfying answer. It's like the childhood thought of, "how do I know you see the same 'blue' as I do?" Consciousness is something I subjectively experience, but I have no idea how I would measure it or validate its existence.

That gets you into some 'eastern' type thinking. Because if you practice any kind of deep meditation, or emptiness meditation, it gets you wondering – where does my sense of 'self' come from. And, am I really separate from everything else?

Which kind of brings things, in my shower-thought mind, to the idea that – perhaps we're being too 'western' about these questions; perhaps the whole universe is conscious? Perhaps things like multiplication matrices can be just as conscious as our chemically triggered neurons?

Anyway, no, none of this is 'scientific' – but, like I said, shower thoughts ...

All this stuff is completely beside the point IMO. It's neither particularly interesting or useful to debate whether LLMs truly are or aren't intelligent -- what matters is what they can do (and for better or worse, it's turning out that the extent of what they can do is mushrooming into areas that had been considered uniquely human).

An analogy would be debating whether digital art is truly art or not, as it's made by pushing around pixels rather than, as had been the case for all of history before digital art, crumbs of pigment or charcoal. I'm sure there was plenty of debate along those lines when digital art was new, but there'd be no point in debating that today: digital art can look just as impressive and allows at least as much freedom of expression as non-digital art, and that's all that matters.

Once upon a time Buridan's donkey was a go-to example for why a truly intelligent entity must always halt. Apparently, the stance on that is now reversed?

Oh, not the halting problem argument against AI, again. Sigh.

There's also the analog argument, that you can't make a human-level AI from digital components, because analog has infinite resolution and can thus handle more data. Analog, of course, has noise, and Shannon's coding theorem applies. Some modest number of bits is sufficient. However, you can sell people who don't believe that expensive HDMI cables.[1]

[1] https://www.pocnetwork.net/technology-news/the-most-expensiv...

If we humans are intelligent, then the analogy could be this: our brains have a finite amount of resources, RAM, the LLM context if you will. So we write things down, reference it if we need to. Vector databases can sort of do this.

Arguing LLMs can't solve AGI by themselves is like arguing an ALU unit will never make a general purpose computer. Yes it's true, it takes more than an ALU to make a CPU and it takes more than an LLM to make _anything_. To gain an intuitive understanding what LLMs can be used for one has to understand how they work. The easiest way I think is the following. Imagine A LLM is a black box and a dictionary. A dictionary contains let's say 65000 words, each word has its number. The input called the prompt is a sequence of numbers. The model spends some time and as its output it gives us not a single number as answer, no, it gives us 65000 numbers. Each number is a probability assignment for each word in the dictionary. The most trivial approach to use this output is to find the word with the highest "probability" value and consider it the output. Then we add that word to the input and we run again until we get the number that means "this is the end". This is how most trivial LLM implementations work.

However, one can be a lot more creative with that output. For example recently there was a paper describing a technique to improve source code generation quality by LLM where the words in the output are compared with various known variables or objects defined elsewhere in the code essentially making it more probable for the model to generate correctly named code items.

Another method can be allowing the model to take multiple paths. For example pick 3 most probable output words, run the model 3 times with each word added to the output, and so on. In the end you have many versions of the output. Feed them back into the model one after another (or together if short enough) and ask it to score them. Choose the best one. To me this is the closest thing we have to "ponder on this for a while" for LLMs.

Edit: I envy people that have access to huge amounts of hardware that can be used to run LLMs. There are so many interesting experiments one can run just by exploring various ways to process model output. It is such an obvious thing I very much doubt it's not being done as we speak by Microsoft, meta, Google. Why are there no papers about it? (other than that paper I mentioned?) Because it's considered "the secret sauce" that gives them a competitive advantage.

There's a difference between being general purpose to a degree and having all of the cognitive abilities and characteristics of an animal like a human. For example, learning online very quickly and efficiently.

https://youtu.be/vyqXLJsmsrk?si=Esxiv601VUhXg1z8

See LeCun's recent talk. He is misjudging GPT, but otherwise his research seems very promising.

See also much of AGI research.

That was a fascinating rabbit hole. I ended up watching his yt presentation of his latest paper. This link is the type of info I still read Hacker News for.

Yes! See basic computer science such as the halting problem, Godel’s theorem, not to mention Hofstader’s book Godel, Escher, Bach. Using a neural network does not repeal the laws of computer science as much as people are inclined to think.

We will all halt.