Hacker News Reader

Comment by anonyfox | Hacker News Reader

I believe more and more that some self-taught software-tinkerer somewhere in the middle of nowhere will have the final idea about how machine learning should work, discovering some simple principles hiding in plain sight. Suddenly, it all will make sense and a hobby-ML-service connected to the internet will start to develop through sheer learning from online resources (forums, ...) into the first strong AI. Probably unnoticed. And then replicate itself through insecure webservers or something like that.

View on HN

Hinton hit on the great idea of using restricted Boltzman machines to pre-train deep neural networks (networks with many hidden layers) and that one idea has changed the field (I sat on a DARPA neural network panel in the 1980s and sold a commercial NN toolkit back then).

That said, I agree that new ideas will likely further move the field along with huge and quick advances. Peter Norvig recently suggested that symbolic AI, but with more contextual information as you get with deep neural networks, may also make a comeback in the field.

The contrastive divergence paper that Hinton published in 2006 definitely set the field off again. I remember entering grad school in 2010 and everyone was still really excited about using unsupervised pretraining. However, nowadays no one uses it.

It just turns out that with GPUs and stochastic gradient descent, no one needs any of that stuff. There are some tricks out there to making it really work, though. In that sense, Hinton's dropout paper has probably had a longer lasting effect on the field.

But either way, I doubt what OP is saying will be true. None of the real advances in deep learning are coming from self-taught coders in the middle of nowhere. They're coming from big labs with lots of resources, both physically and intellectually. This stuff takes a lot of hard thinking by a lot of people who understand optimization and probability. It also takes a ton of compute power and massive datasets, which won't be available to a hobbyist.

I honestly do not think that DL is the answer. It's just a special use case of NN with multiple layers, and NNs itself are just one school of machine learning, IMO not even the most promising one.

What is the most promising one?

The irony is that dropout is increasingly and often viewed as unnecessary in recent deep learning work.

While a hobbyist doesn't have a team of experts (though forums could replace that), right now they do have access to massive data sets and cheap computing resources. There are tons of huge free data sets and cloud and hardware are cheaper than ever.

> Peter Norvig recently suggested that symbolic AI, but with more contextual information as you get with deep neural networks, may also make a comeback in the field.

Could you point me to where he said that? My cursory search came up with an answer on a Quora AMA that was pretty thin on this.

Sorry, I can't find the quote but I will keep looking. There is some talk of structure here https://www.quora.com/What-do-you-think-of-Deep-Learning-2/a...

Edit: also interesting: https://www.quora.com/Are-symbolic-AI-approaches-still-relev...

Charles Stross might not have been far off with the lobsters in Accelerando...

On a datacenter the size of the middle of nowhere.

Reply on news.ycombinator.com