Hacker News Reader

Comment by habitue | Hacker News Reader

I think you have to set expectations based on how much of the ground you're ripping up. If you're adding some layers or some little tweak to an existing architecture, then yeah, going backwards on cifar-10 is a failure.

If, however, you are ripping out backpropagation like this paper is, then you get a big pass. This is not the new paradigm yet, but it's promising that it doesn't just completely fail.

View on HN

This seems to be Hinton's MO though. A few years back he ripped out convolutions for capsules and while he claims it's better and some people might claim it "has potential", no one really uses it for much because, as with this, the actual numerical performance is worse on the tests people care about (e.g. imagenet accuracy).

https://en.wikipedia.org/wiki/Capsule_neural_network

I mean yes, this should be the MO of a tenured professor, making large speculative bets, not hyper optimizing benchmarks

But some of those bets should be right, or else he'd be better spending his time and accumulated knowledge writing a historical monograph.

The backpropagation paper was published in 1986.

It took >20 years for it to be right.

Maybe we ought to give this one some time?

Not sure what you mean by >20 years to be right. I built and trained a 3-layer back-propagating neural net to do OCR on an Apple 2 in 1989 based on that paper. Admittedly, just the 26 upper case characters. But it clearly worked better than the alternatives.

Specifically, tenure is to remove the pressure that "you'd better be right" so professors are free to take meandering tangents through the solution space that don't seem like they'll pay off immediately.

The failure mode of tenure is that the professor just rests on their past accomplishments and doesn't do anything. That's a risk the system takes. In this case though, Geoff Hinton is doing everything right: he's not only not sitting around doing nothing, he's actively trying to obsolete the paradigm he helped usher in, just in case there is a better option out there. I think that's admirable

There is no shortage of paradigms that rip out backprop and deliver worse results.

Also Hinton doesn't have the best track record with his already forgotten/abandoned Capsule networks. I wonder what's the next thing he's going to come up with? He gets a pass because he is famous.

I think it can be simultaneously true that things like this should be tested with toy models we wouldn't expect to do great on CIFAR and also that we shouldn't expect exceptional results just because this person is already famous.

This is so true! But we should keep trying :)

Reply on news.ycombinator.com