Hacker News Reader

Comment by bra-ket | Hacker News Reader

The best part of the article is a quote by G. Hinton, the father of "deep learning", at the end: "My view is throw it all away and start again, I don’t think that’s how the brain works."

View on HN

While Hinton's view need to be noted, I heard a quote attributed to Yann LeCun, something like,

"If you want to learn flying by modeling the biology of birds, you're doing it wrong. Just look at today's airplanes. They have no resemblance to birds at all. Yet they're million times better and faster than any bird."

the problem with that analogy is trying to build an airplane before you figured out the laws of physics.

The physics of lift and aerodynamics were faaaaar from well understood at the time of the first airplanes, though. New areas tend to run a bit ahead of the underlying science; the fundamentals expand to support and improve the applications over time.

but we did have quite a few advances at the time of the first airplane, for example by that time steam & combustion engines were already invented, which required non-trivial understanding of physics, chemistry and material science was very advanced.

I hold a pessimistic view that we are still in hunter-gatherer mode as it comes to understanding cognition.

Well, it's your right to be a pessimist... I tend to think that the current hardware specialized for fast, parallelized linear algebra is at least as good as the wheels available at the start of the industrial revolution, though. We have learning algorithms that can match human/animal performance in a wide - but still constrained - set of tasks, which previous non-learned algorithms hadn't been able to crack. It's a start!

At some point you have to strike rocks to make fire, because the butane lighter hasn't been invented yet. You make do with what's available, and progressively get better at it. I tend to think that we're a couple-few perspective shifts away from getting it 'right,' and that the hardware side likely barely matters. But, I'm an optimist.

People build things without understanding the underlying principles all the time, e.g. the steam engine. You could probably make the case that building things has helped our understanding more than our understanding has helped us build things.

Having said that, you can certainly improve a design when you better understand the fundamentals (vs intuition + trial & error).

"They have no resemblance to birds at all. Yet they're million times better and faster than any bird."

LMAO

A 6 years old kid can see the fundamental resemblance between a bird and a modern passenger airplane: The wings Tail stabilizer Slender body

Planes are faster bigger

Are they better?

Not necessarily, for example, humming bird can fly in a way that is far beyond any human machine in terms of efficiency and flexibility.

Of course man should not imitate birds, because human flight is fundamentally different activity than bird flying. But to say human aviation did not start by mimicking birds, is like to say Ann was not inspired human brain...

I think the point was that aviation at the time did start mimicking birds and that was why there was so much failure. It was not until they let go of mimicking birds and took a different approach that they found success.

See my link to his ICML talk above.

I get what the author was trying to say, but it's still -- a very limited view. Mostly because of the last bit (better/faster).

Birds are to planes, as humans are to cars. Yet can a car leap over barricades, climb mountains, trees, self-repair, turn on a dime, stop instantly, etc, etc?

A plane cannot maneuver like a bird, take off in crazy weather conditions, land on a dime in a tree, stop almost instantly in flight, and change direction, etc.

I think what you've quoted has a lot of value here, for, what we should expect from an artificial brain, isn't a human brain. This is truth. However, while it may be faster in a specific capacity, but it won't have the same characteristics.

So yes, expecting it to be like a human brain doesn't make sense.

Yet better/faster? I don't think we can compare this, they're too different.

(which is really the quote's point, but I just didn't like the better/faster bit at the end...)

Also birds (and insects/bats/pterosaurs) flight is a lot more energy efficient than any plane. Today's deep learning is essentially brute force, burning thousands of watts for anything more complicated which a single human brain can often do in ~15Watts.

The advanced models like GPT-3 are burning millions of watts in the cloud but they're not that much better than what a brain can do (and in many ways worse, as in often requiring supervised learning)

That's the key point. The algorithms need to become more energy efficient to make significant leaps, thus become more like brains.

No, it's not. What's your comparison? Are there birds that can carry 80,000 lb of passenger + cargo weight? Condors fly like fixed-wing aircraft for 99% of their flight, hummingbirds fly more like insects. There isn't one type of bird flight.

This whole HN discussion of bird flight is a trainwreck and reflects massive gaps in understanding of aerodynamics. This is '00s "computer virus news report" level competence in this subject.

We understand the aerodynamics of bird flight, and used it to make fixed-wing planes optimized for carrying lots of cargo. Once we understand the principles behind intelligence, we can make very efficient AI optimized for our usage. But we're still at the point where we don't understand intelligence as well as we understood aerodynamics when building the first planes, so we still have a lot to learn from "birds" - animal brains.

I believe AI will start as a basic principle or idea that can be applied to any sufficiently big state machine that controls e.g. an RC airplane or traffic lights. That idea will be obvious in a hindsight. I'd even make a guess that it will be like a "stateful" state machine that accumulates state in a particular manner and uses that to control the underlying state machine. We still will be nowhere near understanding intelligence, but that clever trick will be enough in most cases.

> But we're still at the point where we don't understand intelligence as well as we understood aerodynamics when building the first planes

Actually, I'd say that our understanding of intelligence is right about at the level of aerodynamics at the dawn of heavier than air flight:

https://youtu.be/Sp7MHZY2ADI

https://youtu.be/gN-ZktmjIfE

I mean, we could quibble about exactly where we are pre- or post-Wright Flyer, but given the amount of AI research that amounts to brute-force flailing about in search of incremental improvements, disagreements on the importance of "biological plausibility" and so on, it's pretty clear that, roughly speaking, AI is currently somewhere in the equivalent of the Lilienthal-Langley-Wright-Curtis continuum (ie. 1890-1910-ish) and still prior to the most important theoretical breakthroughs. IOW, AI has not in my opinion yet achieved an equivalent to aerodynamics' Prandtl lifting-line theory: https://en.m.wikipedia.org/wiki/Lifting-line_theory

Also, birds produce themselves out of an egg, with only food, water and air as production input. They also can produce more of themselves with minimal input. They are also self-repairing/maintaining, something planes cant do.

> single human brain can often do in ~15Watts

There were similar arguments when AlphaGo showed up and beat master Go player Lee Sedol, but is power(in Watt) the right measurement? I always feel like it should be the total energy(in J or Cal) required to transform a computing device like biological brain or electrical computer from knowing nothing to being capable of a skill like Go game. In such sense, deep learning is still more energy efficient than human.

Lee Sedol's lifetime energy consumption for all biological process is around 50MWh, Alpha Go consumed more than 100kW and was likely to have been trained more than 500h, so even counting all the energy Sedol spent eating, dancing or having sex ends up being less than the amount of energy spent to train the machine used to beat him.

> The question of whether a computer can think is no more interesting than the question of whether a submarine can swim. --- Dijkstra

Better/faster we would not directly compare to humans, but to benchmarks and timed experiments.

LeCun is saying to treat "intelligence" the same as "flight" or "swimming". It is a matter of function, not a matter of a specific instantiation on a biological substrate. You don't need to recreate flapping wings to gain "flight", you can strap a combustion engine on a cylinder and beat all birds on earth in regards to speed. You don't say "we don't have flight yet", because an airplane is not able to land on a tree branch. Maybe we don't have yet all the components and aspects of "flight", but this is not a show stopper, and drones have come a long way.

Now the more interesting question becomes: What are the laws of aerodynamics for intelligence?

Aside: I think it is absolutely insane that a conference workshop with papers yet to go through peer-review, is highlighted as a popsci article on VentureBeat. That's such a narrow workshop, that even researchers in the field may be unaware of it. And now these get to read the paper summaries from a HN-story. "the centre cannot hold".

Aside II: Yann LeCun talk from 2019 about this subject (better to debate the source ;)):

> Clearly, Deep Learning research would greatly benefit from better theoretical understanding. DL is partly engineering science in which we create new artifacts through theoretical insight, intuition, biological inspiration, and empirical exploration. But understanding DL is a kind of "physical science" in which the general properties of this artifact is to be understood. The history of science and technology is replete with examples where the technological artifact preceded (not followed) the theoretical understanding: the theory of optics followed the invention of the lens, thermodynamics followed the steam engine, aerodynamics largely followed the airplane, information theory followed radio communication, and computer science followed the programmable calculator. My two main points are that (1) empiricism is a perfectly legitimate method of investigation, albeit an inefficient one, and (2) our challenge is to develop the equivalent of thermodynamics for learning and intelligence. While a theoretical underpinning, even if only conceptual, would greatly accelerate progress, one must be conscious of the limited practical implications of general theories. --- https://www.ias.edu/video/DeepLearningConf/2019-0222-YannLeC...

See my link to his 2013 ICML talk above. There's a very nice photo of L'Avion III de Clement Ader, a plane modeled as a bird.

The better/faster part is my embellishment :)

See my link to his ICML 2013 presentation above.

It's true we don't build planes like birds. But we understand how bird wings work, and we understand why we don't design planes like bird wings.

We don't have the same kind of understanding of how brains learn, so the comparison is not quite right.

When we understand how to build things that learn like brains, we'll be in a better position to say things like "Ok this is strictly worse than backprop, let's stick with backprop" or "Actually, this is better than backprop because X", (or, more likely, there are things we can use from both). Until we have that understanding it's silly to stop trying to understand how the brain does things.

That being said, nobody is going to stop working on backprop, and no one is going to stop working on understanding biological mechanisms . Research works by a bunch of people investigating different avenues simultaneously.

Reminds me of Dijkstra's aphorism:

> "The question of whether machines can think is about as relevant as the question of whether submarines can swim."

I found the source. He and Ranzato gave a talk at ICML 2013

https://cilvr.nyu.edu/lib/exe/fetch.php?media=deeplearning:2...

Slide #9

Let's be inspired by nature, but not too much

It's nice imitate Nature,

But we also need to understand

For airplanes, we developed aerodynamics and compressible fluid dynamics.

Question : what is the equivalent of aerodynamics for understanding intelligence

I would caveat "million times better." When 747s pop out baby 747s and procreate and refuel autonomously for 60 millon years I will give you million times better.

But I like the gist of the quote.

Better is subjective. A specific AI may perform better than a human (eg. processing images for hours, faster and longer than a human) But we're still not able to have an AI you can talk to like you talk to a human (gpt-3 is still nowhere enough in my book, albeit being a good technological feat)

The main problem is not backpropagation though, but the fixation of resources on DL projects (that's what I call local minimum!). In my department, for example, they don't seem to care about the application, integration, deployment etc, as long as it's DL or DRL.

Didn’t he recently publish an article about a drastically different way to approach machine learning called capsule networks?

There's nothing "drastically different" about capsules networks.

Reply on news.ycombinator.com