Gary Marcus discusses AI's technical problems
cacm.acm.org
cacm.acm.org
How can anyone think he's arguing in good faith at this point. That essay was published after gpt3 prior to gpt 4 - and he's claiming it was correct!
GPT-3: Useful as autocomplete. Still error prone, but vastly better than any pre-AI autocomplete
GPT-4: Already capable of independently coding up simple functions based on natural language.
O3-mini: Can code in say top 5% of codeforces.
There's a 2 years gap between each of them.
More over, intelligence has a superexponential return, 90IQ->100IQ < 100IQ->110IQ in terms of returns.
That's the second time I've seen the claim that linear increases in intelligence have "superexponential" results, first time was Altman's blog.
But I've not seen any justification for this.
(As you specifically say IQ, note that an IQ is defined as a mapping of standard deviations rather than a mapping of absolute skill, the normal mapping is 15 points being 1σ).
Gary Marcus didn't make a lot of specific criticisms or concrete predictions in his essay [0], but some of his criticisms of GPT-3 were:
- "For all its fluency, GPT-3 can neither integrate information from basic web searches nor reason about the most basic everyday phenomena."
- "Researchers at DeepMind and elsewhere have been trying desperately to patch the toxic language and misinformation problems, but have thus far come up dry."
- "Deep learning on its own continues to struggle even in domains as orderly as arithmetic."
Are these not all dramatically improved, no matter how you measure them, in the past three years?
[0] https://nautil.us/deep-learning-is-hitting-a-wall-238440/
It's very difficult to understand this statement. What meaning of "qualitatively" could possibly make it true?
Underneath it all, there is some hope that an innovation might come about to keep the wave going, and indeed, a new branch of ML being discovered could revolutionize AI and actually be worthy of the hype that LLMs have now, but that has nothing to do with the LLM craze.
It's cool that we have them, and I also appreciate what Stable Diffusion has brought to the world, but in terms of how much LLMs influenced me, they only shorted the time it takes for me to read the documentation.
I don't think that machines cannot be more intelligent than humans. I don't think that the fact that they use linear algebra and mathematical functions makes the computers inferior to humans. I just think that the current algorithms suck. I want better algos so we can have actual AI instead of this trash.
It also only affects those with a "weak immune system" i.e. those whose bullshit filter doesn't function.
AI is here to stay for some tasks (segment anything, diffusion image generation for accelerating certain kinds of Photoshop), but LLMs are a dead end and AI Winter 2 is coming. They don't work for programming or law or medicine or mechanical engineering or even writing most emails because it's faster to just write the email yourself than to ask the AI to do it.
While writing this, it occured to me that he would get even goose bumps at reading this comment because it, after all, I am giving him attention.
My impression is the opposite: I would describe Gary Marcus as having all his opinions perfectly aligned to a singular viewpoint at all times regardless of weather (or evidence).
Depending on how he can get interview or get seat at the table, he may chose exact opposite of positions.
He has an infinite quantity of time and somehow always finds someone to interview him about the fact that AIs don't work and have been hitting a wall since the last ten years. He invents some metrics that nobody has agreed on, and says that they're below that.