Deep Learning: A Critical Appraisal [pdf] | Hacker News Reader

Deep Learning: A Critical Appraisal [pdf]

arxiv.org

85 pointssarosh18 comments

> deep learning must be supplemented by other techniques if we are to reach artificial general intelligence

I don't think anyone major ever disputed that.

Having said that, thousand times yes to the author's concerns. Deep learning is AI's cryptocurrency in terms of being overhyped, although its main proponents are not to blame for that.

Why do you think it's overhyped? It pushed state-of-art results in quite a few difficult domains by quite high margin; it's deservedly praised. Or you don't like that you can't really understand what is going on inside, despite it using simple math and primitive non-linear optimization, making it "conceptually" inferior and not as "tasty" to other ML methods where we can actually prove something?

Deep learning is especially overhyped by articles that describe it as "just like the human brain". It's a useful technique that lets you skip lots of feature engineering by just letting the classifier learn its own features, but it is also really easy to project abilities onto neural networks that they don't really have.

Deep learning is not magic, for every network architecture that beats the state of the art, there are a hundred very similar ones that completely fail, or run too slow, or don't fit into GPU memory ... and the only way we know to get improvements is to fiddle with the hyperparameters until everything works out.

Because today, most lay people equate deep learning with AI.

The black box part is also an issue which both the author of the paper and literally everyone else, including Google's Peter Norvig, is concerned about. But that's not related to the hype part.

Deep learning has had much greater impact within AI than cryptocurrencies have had in cryptography/systems/security

I should have added, "what cryptocurrencies are to fintech". Is it better :) ?

To get AI you have an interface problem you need to solve since reality is a bitch to parse; this is just the start, what about the formulation of goals? How would you even make that in code? You would need to start with a certain kind of data that was extracted by the sensors but how will you ever know you have the correct set? You have two problems to solve simultaneously getting the right data and building the right system on top of it -- this core data when done in reality is not the same core data that would make this possible in a simulation and the system on top while maybe having the same fundamental form is also very likely different. Makings intelligent agents in a simulation is not the same as making intelligent agents arbitrarily useful in the real world, granted a simulation is a good means to explore techniques -- in a simulation I control the sensory apparatus of the agent perfectly. So I think the lesson is you either build systems from which it is easy to stream data and configure programmatically or deal with the fact that this is likely impossible without simulating a universe to high fidelity. Deep Learning is extremely useful as it is, not the same as cryptocurrencies which are useful but this usefulness will never be properly realized due to human nature -- right now their primary purpose seems to be to satiate greed and humans with poor impulse control while promising a lot of outlandish things.

You hit the nail on the head.

Oren Etzioni has been shouting about the knowledge representation for years (full disclosure: this is something I'm focused on, too).

Why do we want to reach AGI? There are around 7.5 billion biological general intelligences, and we already know how to make more of them. Is it to save companies labor costs and increase their profit margins? I'm not sure that's ultimately in the interest of most people, or even the long term health of the economy.

Seriously, why do we want to replace human intelligence? I get augmenting it with narrow forms of automation, but AGI is a different animal.

> I don't think anyone major ever disputed that.

Is it that obvious though? Any good evidence that human brain on the planet scale cluster can't be modelled with DL?

This is somewhat of an opinion piece. We need more articles like it to counterbalance the "AI is the new electricity" crowd. Hyping deep learning isn't healthy.

Almost all concerns in the paper are active research topics and do have certain solutions which do use some sort of deep learning approach. Depending on the viewpoint and interpretation, you could say that some of these approaches are hybrid solutions, but this is really just a matter of interpretation. No-one is really denying that the stated concerns are valid concerns. But also, no-one would say that the current knowledge gained from deep learning research will not be useful in the future. Of course, maybe for some aspects, you would need more radical new ideas, but I doubt that for future methods, nothing from the current methods will be used in some way.

E.g.:

3.1. Deep learning thus far is data hungry. First, you could argue that on a low-level, an animal/human gets quite a lot of visual and audio input, so it's data hungry as well. Then, you could argue that the evolution did already do some sort of pretraining/pre-wiring which helps, using million of years of data. Then, related to this is the topic of unsupervised learning and reinforcement learning. Then, dealing with the aspect of learning with small amounts of data, there are the active research topics of one-shot-learning, zero-shot-learning of few-shot-learning. Related is also meta-learning.

3.2. Deep learning thus far is shallow and has limited capacity for transfer. Transfer-learning, meta-learning and multi-task-learning are active research areas which deal with this.

3.3. Deep learning thus far has no natural way to deal with hierarchical structure. There are various approaches also for this. This is also an active research area.

3.4. Deep learning thus far has struggled with open-ended inference. This is also an active research area.

3.5. Deep learning thus far is not sufficiently transparent. Also this is an active research area. And then, you could also argue that the biological brain also suffers at this.

3.6. Deep learning thus far has not been well integrated with prior knowledge. This is also an active research area.

Etc.

In some of those cases, the active research has been going on for as long as deep learning itself- for instance, one-shot-learning comes from the '90s, if memory serves, so does transfer learning ('93, wikipedia says). My hunch is that in such cases only mediocre solutions exist.

And of course, just because there's reasearch in a given area doesn't mean that progress will necessarily be made. Frex, research on semantics has been going on since the dawn of AI and we 're not even close yet.

Personally, I think it's always good to have people pointing out limitations of a technique. Minsky and Papert caused a lot of consternation back in Perceptrons, but without that, who knows when the ANN researchers would have gotten off their butts and tried to solve real problems.

More context here: https://news.ycombinator.com/item?id=16083325

Different perspectives and research backgrounds converging to the same limits of the given tool is very good for defining a boundary while containing the hype. More in general, it still seems generally inefficient (and very risky from a regulator point of view) to deploy full AI agents in dynamic, human, imperfect environments, eg. self-driving cars in the common traffic flow.

This isn't a great paper (as you can tell by how often the author cites himself).

It isn't really worth responding too - it's either attacking claims which are never made, or so outrageously wrong it appears to be trolling.

Most academics cite their own work... That’s not a quality marker, it’s the norm.

Care to give more info on your second paragraph?

Can you give examples where the paper is wrong? In large things, not nitpicking. I'm just genuinely curious.

Reply on news.ycombinator.com