Geoffrey Hinton spent 30 years on an idea many other scientists dismissed

I remember that in the early 90s, neural nets were supposed to be the huge new thing. Many scanners shipped with built-in OCR, the Apple Newton had handwriting recognition on a PDA, and my Centris 660AV could do speech recognition & text-to-speech out of the box.

But they ultimately weren't powerful enough to satisfy customers or meaningfully change how people interacted with computers, so they failed in the market, and the hype cycle moved on to the World Wide Web.

I guess this shows the power of continuing to study something when the fad goes away, so that you're well positioned to capitalize when the next fad hits.

As a counterpoint, one of the most successful (classes of) machine learning algorithms are Decision Tree learners, whose models are decidedly symbolic.

There's also plenty of work on learning first-order logic theories with neural nets (see for instance the work of Artur D'Avilla Garcez).

The problem with rules is that it's hard to develop and maintain large rule-bases. However, it's perfectly possible to do machine learning for rules -and logic-based machine learning is totally a thing (full disclosure: it's the thing I'm doing a PhD on). You probably haven't heard of it though because data scientists tend to be good with statistics but bad with symbolic logic.

AI has had a lot of fads, but neural networks are here to stay, in my opinion.

I have just finished reading this interesting wiki titled AI winter defined as a period of reduced funding and interest in AI research:

https://en.m.wikipedia.org/wiki/AI_winter

There have been such winters since the 60's.

I think that is what the article may be getting it.

https://www.semanticscholar.org/author/Geoffrey-E-Hinton/169...

Citations to his papers have been rising steadily from between 88 and 107 in 1987 to between 685 and 826 in 1999. That's hardly an unpopular researcher.

And for a bit of comparison with other machine learning researchers, here's a link to a data set of family relations from a 1986 paper by Hinton:

https://archive.ics.uci.edu/ml/datasets/Kinship

At the bottom of that page, in the Relevant Papers sections there's two links to two papers using the data set, one Hinton's own paper that introduces it and one by Quinlan.

Clicking on the [Web Link] links for the two papers, I can see the references to those papers. There is a single reference to Quinlan's paper. There are 43 to Hinton's, of which all but 6 are from 1999 and earlier. And those are not self-references, neither references by Bengio, Le Cun et al. If there is a clique, it is hard to see it.

So there was a lot of interest to Hinton's work even in the years he was supposed to be "exiled to the academic hinterland" as another article said.

Truth! I happened to go through grad school at a time when SVMs and kernel methods were cool, neural nets were the opposite, and Hinton and his students were oddball outcasts at NIPS. Kudos to Hinton for sticking to his story until the current "deep learning" hype wave presumably allowed him to buy a yacht and an island. I imagine we'll be hearing about some other non-linear optimization technique in a few years.

"What the book does prove is that in three-layered feed-forward perceptrons (with a so-called "hidden" or "intermediary" layer), it is not possible to compute some predicates unless at least one of the neurons in the first layer of neurons (the "intermediary" layer) is connected with a non-null weight to each and every input. This was contrary to a hope held by some researchers in relying mostly on networks with a few layers of "local" neurons, each one connected only to a small number of inputs. A feed-forward machine with "local" neurons is much easier to build and use than a larger, fully connected neural network, so researchers at the time concentrated on these instead of on more complicated models."

Neural Networks weren't really thought of very highly until CNNs started winning image recognition competitions in the early 2010's.

I think most people had the feeling that they were interesting tools to learn how the brain worked, but too slow and opaque to be practical statistical tools. I've been following Hinton for a while (because of Hinton and Shallice 1991), and my understanding is that it was really hard for him to get funding especially when he was just starting out.

The fact that so much of the work from the mid-80's to 2000 came from just a few labs should tell you how hard it was to get funding for that kind of research.

When I did my M.S. reasearch in 2010 SVMs were far more popular than neural networks. Deep learning was just over the horizon and most people had given up on it being useful for learning real world problems. We had proven that NNs could learn anything given enough data and the right structure, but the hard part in real life was having enough data and finding the correct structure. So SVMs seemed more robust with available datasets, and most people used those. There were/are several other popular methods, like nearest neighbors and tree-based methods, and NNs just weren't as sexy as they seemed in the late 80's.

If you took half the apps built today and tried to run them on compatible hardware from 20 years ago, they would all flop.

Without the understanding of modern computing, one would have dismissed all such inventions as useless wastes of time.

Historically, almost all ML approaches were based on separation in Euclidean (vector) space, which is understandable because they were developed for much weaker computers. However, really useful ML tasks require to deal with huge nonlinearities, and the fact that you're training in linear space becomes less relevant.

Neural networks have surmounted the nonlinear difficulty by increasing the number of layers. But it's a question whether a similar result couldn't be achieved with Bayesian networks on binary representations (which is approach I favor).

There is some evidence that the precision of linear calculation in neuron (for example, resolution of the weights) doesn't really make much difference in the performance of neural network. Could it be that the neural learning through vector manipulation is just an artifact of the origin of the neural networks, and the really important thing is the overall organization of the network (the layers)?

Geoffrey Everest Hinton → Howard Everest Hinton → George Hinton → Mary Ellen Hinton → George Boole

It certainly seems plausible: https://en.wikipedia.org/wiki/George_Boole

So why isn't this something that's in the realm of possibility for computers?