Hacker News Reader

Comment by garymarcus | Hacker News Reader

Actually, I haven’t changed my tune on this particular question since 1992 (link in the article). deep learning alone is not now and never will be the right path to strong AI. no need to change my mind about that…

ps thanks for posting the article!

View on HN

>I haven’t changed my tune on this particular question since 1992

Well DL certainly hasn't been against a wall since '92.

Pulls out phone with highly accurate voice to text, predictive keyboard, sensor activity detection, auto-categorization/labeling of photos, facial recognition, learned speech synthesis etc etc (thats just a few just in the consumer space... with no mention of government/commercial/scientific applications)

They had all that in 92?

Yes. In a sense. The sigmoid/neural network was well conceived, the major limiting factor was hardware and training data.

ML isn't really that much further I'd wager. We've just finally gotten the computer yo catch up enough we can spit out a handful of reasonable domain specific function simulators.

We're no closer to a feasible integration thereof to the point of emergent consciousness.

>to the point of emergent consciousness

But that isn't the goal nor the argument being made (not to rabbit hole in discussing that consciousness isn't even really a scientific term that can meaningfully be applied).

We had some mathematical notions yes... and we have made a ton of progress since then. The perceptron doesn't hold a candle to methods today, not even close... though yes it is a building block for the field. I don't know how that could be all that controversial.

He's really talking about a path towards AGI, not ML being able to do many tasks like this. There's work towards ML doing causal inference, but CI has been a major challenge for Deep Learning (a specific type of ML) and is likely not possible with it alone (see reference to hybrid models). Of course, if he were saying that ML/DL hasn't improved significantly in these recent decades then yes, he would be being dishonest. There even has been work in explicit density models, symbolic manipulation, and causal inference. All things he (presumably) cares about. But these things don't get nearly the hype nor the research power and thus is a lot slower. In the end there's really two camps. Those trying to do things and those trying to build models that understand. But note that he's using DL as a specific term and not in place of ML nor AI.

Yes, "they had all that" in 1992: speech recognition, image classification, speech synthesis etc. These are problems as old as the field [1]. They didn't work as well as they do now, but computers where not as powerful and datasets where not as large, as they are now. The approaches though, that do all these things today, are still the same old statistical learning approaches, just with more data and more compute.

At the same time, we don't know if what statistical approaches can't do, but symbolic approaches excel at, like reasoning, for example, would also benefit from the modern advances in computational power, because there's almost nobody trying. All the large tech corporations are head-over-heels for statistical learning and most people are running behind them, following the current trend. So nobody's trying to scale up, I don't know, classical planning or SAT solvers, to the extent that Google, Facebook et al, have scaled up neural nets.

_______________

[1] Specifically, the field of pattern recognition, which is older than machine learning as a field of research. See the Introduction chapter in "Statistical Learning Theory" (the textbook) by Vapnik for a quick run-through of the history of statistical inference, pattern recognition and statisical learning.

I've enjoyed your discussion of AI quite a bit over the years, especially your comments on the senselessness of GPT-3.

That said, it seems to me that "possibly hitting diminishing returns" would be a better phrasing of the situation. Google's Alpha Fold is considered a serious advance in the field of protein folding. Deep learning has aided astronomers find a variety of things. etc.

Disregarding the negative comments here, I found the article to be very much in line with my experience as a scientist in a different field working with deep learning for solving practical problems (reducing compute needed for physics, PDE constrained inverse problems) for a few years. I am not a deep learning detractor, nor am I a fan boy. Increasingly we have found that using hybrid methods: deep learning to extract parameters from large amount of data but then using the said parameters in a physics based method task executor guards against nonsense results. Small number (and dimensionality ) of the deep learning extracted parameters help with reasoning on "is deep learning still working as expected with this data" and continuing from there is usually safe. Even when the whole workflow is deep learning based, we have had to clearly reason out the domain, range and form for activation functions for critical layers in the DL network to make it work. No amount of throw in a huge resnet, transformer, FNO, chimera would do the trick without conscious thought on what the network is supposed to do. I would argue a lot of useful deep learning in hitherto un explored applications will need to have the symbolic manipulations encoded in the structure of the network.

> deep learning alone is not now and never will be the right path to strong AI

Would you argue that “if not (deep learning, x), then not artificial consciousness” where x is any other computational technique?

Curious, what is then?

No one knows. We need to do more basic research to figure that out.

My guess is that it will take large scale quantum computing. But that's just speculation, I don't have any proof.

brains in jars.

Nobody truly knows, but as an outsider what's really infuriating for me is the tunnel-visioned nature of ML/DL research and investments. Especially money-wise, practically no other techniques are getting even 1% of the funding so it seems that ML/DL is some sort of a local maxima that we can't seem to crawl out of.

IMO the path to a better AI very likely is tightly bound to ML/DL but to me it's obvious that they by themselves are not it. It's very likely a combination of techniques, ML/DL included.

Reply on news.ycombinator.com