DeepMind to Work Directly With Google’s Search Team

Then it ends up taking days.

We all think of incredibly futuristic sentient robots, but what we should really be worried about are the implications this acquisition will have on Google Search, a tool that is already used by millions of people. A tool that millions of people already superimpose to "the Internet" itself. A tool that already has a strong and unmitigated influence on people choices.

That may not be completely clear in the U.S. Market, but take a stroll in peripheral markets and, to quip the famous commercial, you'll see why 2014 already looks a lot like "1984".

He then started accomplishing it within three, if the "Playing Atari via Deep Learning" paper is to be believed.

Now, as far as the difference between PCA and eg an autoencoder (which is similar to an RBM, but different--both RBMs and autoencoders are "deep neural networks")--well, PCA just learns a linear mapping. You can stack PCAs on top of each other I guess, but it is functionally equivalent to a single layer. So it makes certain assumptions about the behavior of the data--specifically, that it's linear.

A neural network layer calculation looks like this: nonlin(Wx + b) A deep neural network forward pass looks something like this:

W3 .* nonlin(W2 .* nonlin(W1 .* INPUT+b1)+ b2) + b3)

Nonlin can mean a variety of things. Usually tanh, sigmoid, or max(0,x). If you remove those nonlinearities, it basically becomes like PCA, like a model that learns a linear mapping. When you have those nonlinearities, you get a non-linear mapping which, theoretically, lets you address a wider variety of problems.

But yeah, if you want to learn about deep learning, just learn about neural networks. It's the same thing.

Also, re: your point on local minima--on the upper end of experiments that have been done, you've got over a billion parameters. 10 million parameters would be on the relatively small side. I think it's reasonable for there to be a variety of possible solutions to any given problem, and the whole idea is that we're using models that are very flexible due to their high capacity. The problem historically has been using the capacity effectively. However, regularization methods like dropout help a lot, and I've found it's easier to think of neural networks as "feature learners". Each one of those neurons is theoretically a feature its learning. And even still, if you look at the features that are learned by convolutional nets (which are easier to visualize due to their being used on vision problems), they tend to arrive at similar features even though it starts from fairly different initializations.

This is only interesting if you want to analyze the model. Not if you just want a function f(x) that tells you whether x is a cat or a dog.

As for Google's KG, updating the model in real time isn't really necessary, knowledge doesn't typically advance emails or blog posts at a time. They already have that in their search, knowledge is something that is mutually agreed upon and takes some amount of consensus. Using the structured sources (like freebase) is much better than trying to do dependency parsing on some blog post and try to make sense of it. Powerset tried that and it didn't work then, and won't work for at least a few years. ML advances not in leaps and bounds but rather by little pieces of the puzzle falling into place. The biggest thing to happen to the ML world isn't algorithms or fast computers, but the abundance of data available. Like pagerank, the more data it has, the better it works, the same is true of ML systems. Google has more data than god, and it is through this they will push the frontiers of AI - along with IBM and the others working in this field.

The thing I noticed is that DeepMind is good at advance game AI, smarter recommendation system for online commerce and image search, not context or knowledge with distributed system. Is it going to deviate DeepMind's direction? This is my concern.