Cargo Cult AI

This claim seems over general, because you can ask gpt-4 'Why' and 'How' questions and it seems to do a pretty good job.

The author doesn't provide a lot of contrary evidence.

There's so many articles saying "LLMs can't do X" that leave me wondering whether the author has even tried. Maybe they've tried and have some more sophisticated argument, but I often don't see it.

If I was going to knock LLMs for being unable to do basic science, in particular, I'd make sure to do some experiments first!

This doesn’t require intelligence. The Oblique Strategies card deck does a similar thing, and you’re drawing randomly from a deck of cards.

AI-provided hints are considerably better guesses than a deck of cards can give you. When writing code using GPT4 and Code Interpreter, it often guesses right the first time about what the bug is in some code.

Other times it will guess wrong, repeatedly, so I need to undo and tell it more specifically what to do.

Does GPT4 understand trigonometry? I went back and forth on this. Yes, it does! But no it doesn’t, it just got lucky. But yes, it does, if you ask it to explain the math first. But no, it doesn’t, it got lucky again.

What seems clear is that it “understands” context very well. It will make guesses that are entirely appropriate when writing code that does calculations using trigonometry. It will come up with plausible explanations, given that we are doing trigonometry. They still seem to be guesses, though.

In the short term, AI chatbots don’t learn, cannot learn, because they implement a pure function. When you start a new chat, the state is reset. Instead, we learn to give them better context and get better results. The learning is in the prompt engineering.

As the article states, there's no understanding, "just" a regurgitation of sufficiently-trained language (which is utterly astonishing, don't get me wrong). So how does an LLM-AI turn into an AGI? Can it? Should it? How do we tell?

What should be a 21st century Turing Test? Perhaps giving an AGI-candidate a novel situation and seeing how it reacts. I think that trying to apply a rigorous scientific method to such a test might end up being as "messy" as psychology is viewed as being today. It would be quite gratifying if we really did need Asimov's Susan Calvin to tell jokes to Johnny 5!

I also find the prospect of an utterly incomprehensible AGI really interesting in that maybe intelligence isn't only possible via a big messy blob of neurons soaked in a chemical soup of weirdness. Maybe there's a regimented, strict path towards an intelligence that surpasses us in every way?

Does anyone else think we'll be unable to recognize AGI when we see it?

[1] A little column a, a little column b[2]. It's a gut feeling. Will AGIs have guts?

[2] Aren't footnotes great? I still think HN needs more markdown support, but I wouldn't want to be the one to implement it!

You don't get to hold up Newton and Kepler etc and say that their level of scientific insight is the bar for "intelligence" unless you're willing to say that almost all humans, who have not extracted some beautifully simple and concise mathematical rule underpinning the noisy data of our experience, are not intelligent. Feynman, in his original cargo cult speech, pokes holes at broad sections of academia in saying they aren't sufficiently scientific, including a physics example where an experiment is compromised by re-using data from a different apparatus b/c they couldn't get accelerator time for the "boring" half of it. Are all the physicists involved not "intelligent" because they don't rise to Feynman's high standards?

There are lots of good questions to ask about LLMs, reasoning and intelligence -- but this really seems like it's just goal-post moving to "intelligence is what's displayed by the very best scientists, not 99.999% of humans."

I think many of us would like our consciousness to be rooted in quantum noise, where it's possible to imagine ourselves having "free will", whatever that is, or anything that doesn't make us completely deterministic machines with a helpless observer.

Unfortunately I can't find any evidence of that, but I do know about people who suffered the loss of all but their short-term memory, and they react so predictably to the same sensory inputs (even with some variance in the input) that it's spooky.

Using current LLMs as an example, you could argue that they are a lot more nondeterministic than humans as it's normal to add some "random" values iirc.

Obviously, LLMs are a specific type of AI that do not emulate many of the characteristics and capabilities of animals like humans.

However, they absolutely do have general-purpose applications and the leading edge ones can reason, although in some ways not quite as robustly as some humans at this time.

But one core aspect of "AGI" used to be distinguishing general purpose versus narrow AI. GPT-4 clearly has general purpose utility.

So part of what has happened has just been goal post moving. The other aspect of it is a failure to recognize that there are a variety of cognitive abilities or characteristics that go into animal (human) cognition rather than just one thing.

LLMs simulate some of the language abilities but not other things like stream of subjective experience tied to high bandwidth sensory experience, emotions, certain types of very fast adaptation, properties of life, self-preservation and reproduction, advanced spatial and motion systems, etc. There are a number of cognitive characteristics that animals and/or humans have which are entirely missing from LLMs.

However, again, it's clear that LLMs, multimodal LLMs, and similar systems, do have general purpose application in a way that many may have previously assumed only a more complete emulation of a human would require.

Within a decade or two, when we have hyperspeed superintelligent AI, we may find out the hard way (and maybe too late) that not having all of those animal characteristics in AI systems was a _good_ thing.

This answer is wrong, and it is the reason Kac's famous question merits being revisited in the context of today's AI to solve complex problems. In the 1990s, mathematicians finally proved that it is, in fact, not possible to hear the shape of a drum, or at least not uniquely.9 This is because drumheads exist of different shapes that produce exactly the same sound, or in mathematical terms, are isospectral.”

That seems entirely unfair. The question is if you hear the shape of a drum. The actual answer is yes. But he says this answer is wrong, because you can’t uniquely hear the shape of the drum.

I’m trying to decide if this goalpost moving is a better metaphor for this full piece, or for much of the analysis of AI in general.

Hey! I thought we're not supposed to get political on HN!

I’m convinced the author has never used an LLM before.

It is pretty good at what it does. Why can’t people just appreciate that?

I have found it really good at brainstorming, generating code, outlining, rephrasing on command various text, etc.

So are majority of humans it seems.

Assuming he would not be promptly hired by some FAANG to boost their AI prestige, he would likely proceed to sharply criticize the cargo cult AI movement. He would invent a few new classes of neural networks for breakfast but crucially, he might construct a meta-theory about how, when and why something based on least-squares can convincingly fit the structure of a corpus of human generated information artifacts.

The article does a good job to review of the various loose arguments [1] which point to the vast difference between human intelligence and least-square type models.

It is unlikely to move the debate forward though. AI "ultras" have developed a sophistry that negates any intelligent discussion. It consists of repeating ad nauseam: the brain is an input-output machine that learns on data, software is an input-output machine that learns on data, ergo, software = brain, and ergo, it is only a matter of (unspecified) time before super-software becomes super-brain.

[0] This stance does not reduce their utility, it just removes the metaphysics

[1] They are loose because we know so little about how the higher functions of the brain come about and work. E.g., when Gauss invented the least-squares method something remarkable happened in his brain that is unlikely to have much to do with least-squares. The fact that not every human is Gauss does not mean that this capability is not innate and latent in the structure of all human brain.

  It is worth noting the major difference between two types of models: those used in AI and those encountered in theoretical physics.

Was Urek, the first caveman able to generalize a physical theory of orbital motion when he looked up at the sky? It took humans millions of tries and hundreds of thousands of generations to come up with Kepler. Why does he give up on NNs so quickly?

But today's models do ask and answer those questions.

The author may be confused by the fact that the UI of most systems is currently set up to be driven by human prompts, but this is nothing to do with the nature of the AI model; it's more of how the system is integrated into its environment. We can easily choose to give these systems their own initiative, with much (even) simpler programming than was required to make the AI models in the first place.

We can get into a semantics argument about whether it's the AI model that has the initiative, or the runloop driving it, but that's missing the point that what matters is the total effect of the entire system, not whether you can point to some subcomponent of it and say that component by itself doesn't meet the definition of whatever term (consciousness, reason, etc.) you are gatekeeping.

A lot of LLM criticism is like that joke. People seem to complain too much about LLM not being what they want it to be without realizing that they are indeed much better than what we could have built 20 years ago.

If LLMs can be proven to add economic value that is good enough. They are in themselves useful with all their limitations. If they are profitable people will invest more in them and over time we will have better understanding of AGI and other aspects of intelligence.

When I showed my simpler approach they were confused... What do you mean by linear polynomial function?

The author lost me here. GPT-4 regularly asks me these questions, or sort of. This depends on your prompting. GPT-4 can understand when you are approaching a problem from the wrong side, and suggest a different and more correct approach.

The problem is the default prompting. GPT-4 will be biased by your prompt and tries to go along with it even if it internally ("knows") or has biases against it.

Doesn't matter really, even with this limitation AI models today can produce extraordinary value for businesses. AI's are useful today. I don't see how this is even contested, when it's empirically true.

One ancient debate in AGI (greatly simplified here) is whether intelligence can emerge from brute force computation alone (statistics) or from sophisticated algorithms (logic).

I certainly started on the logic side. For example, it seemed that if we could create an algorithm or data structure to encode (and retrieve) arbitrary knowledge, then we'd be well on our way to intelligence. Thus all the work on logic programming, symbolic representations, and efforts like Cyc.

But by the time Deep Blue beat Kasparov in NYC--something that many people thought impossible--there was evidence that brute force computation was enough.

Chess is a great example because humans do not play chess like Deep Blue does. Humans play chess by pattern matching and not by exhaustively generating move trees (brute force). For a while it seemed like true intelligence depended on pattern matching, and that brute force was a neat parlor trick but ultimately a dead-end.

But LLMs are brute force pattern matchers. Instead of encoding the rules of grammar, we just feed it a trillion tokens and let it find the patterns. The amazing thing is that you can get intelligent behavior and even creativity just by pattern matching.

In fact, think about what creativity is. Creativity is coming up with novel solutions satisfying certain constraints. How do humans do that? Individually we do it by coming up with lots of different solutions (sometimes only as fleeting thoughts) and testing them to see if they satisfy the constraints. Even when humans get a "flash of inspiration" it's often because their subconscious is busy testing various ideas.

Collectively, humans invent and discover by having the best ideas, the best art, the best scientific models, win in the competitive marketplace of ideas. But it's still just trial and error. I think that's the genius of the scientific method: that it takes many ideas--some good, most bad--and keeps only the best. But it's not magic. It's just a process.

In my opinion, the article is using old arguments against AGI ("computers cannot be creative") exactly when LLMs are exhibiting the very traits that they say can't exist. LLMs are absolutely exhibiting "creativity". Maybe they are only as creative as an average human and not a genius. Maybe they're only as creative as a five-year-old.

LLMs are forcing us to confront the fact that intelligence isn't some magic force or elusive unobtanium--it's just the product of brute force pattern matching. It's like when Copernicus postulated that the Earth is not the center of the universe. It's bound to upset people.

  How much more improvement can really be eked out of this approach on the pathway to AGI, and is it really sustainable?

Forget all this highbrow stuff, my still intact test is, can something walk into any random house on the street, plumb a toilet and brew a cup of coffee. Moravec was right in the 80s when he pointed out that the easy things are hard and the hard things are easy. Simulating human cognitive abilities primarily exists to satisfy human testers, it doesn't say much about general intelligence.