Deep Learning 2016: The Year in Review
deeplearningweekly.com
deeplearningweekly.com
I'm really glad this was the approach taken and his examples really show why these are the important metrics with respect to DL. It's hard to underplay the dependencies between these three frontiers. Radical breakthroughs in algorithms make it easier to get better results on existing infrastructure. Better data on the same algorithms and infrastructure can transform results, and on and on.
- Weather Prediction
- Stock Market Prediction
- Healthcare (Better prediction, better reading of diagnostic tools)
- Poker
I can assure you this is being done. Anything that works is kept secret.
The assumption that only when we have comparable number of parameters to the number of brain cells we can have nearly human performance is no longer true. It gradually loses traction once people discover that stochastic connections may be more effective than merely increase layers/parameters.
Brain dead trend-extrapolation is a pretty good prediction tool, but I suppose my assumption that modern NNs aren't more efficient was sort of an arbitrary assumption that makes the exercise useless. Is there any work on this?
I'm not sure what evidence the performance of these systems show us, as bees are pretty smart for their size. These system are about as optimized for lip-reading and translation as bees are for their various tasks, so it wouldn't be surprise me if it turned out that they are similarly powerful per synapse to NNs.
http://timdettmers.com/2015/07/27/brain-vs-deep-learning-sin...
A key assumption I don't share with Detmer is that we must match the brain's raw computational power to have a chance at human-level AI. From an evolutionary point of view, most of the brain's computations surely must be superfluous to intelligence and necessary instead to a vast host of other biological functions.
I'm not sure how you get that. My point is that artificial NNs may well be more efficient than biological ones because they are punching so far above their weight based on what you would expect from a naive comparison of parameters, in which case it's not 17 iterations but a lot less. Further, modern NNs don't typically max out the memory of a single GPU (since we want to do minibatches of at least n=10 for the training phase and researchers care a lot less about the forward passes or deployment), and groups like Google Brain or Deepmind have shown the ability to use ~1000 GPUs, so that's 4 orders of magnitude right there (doable with asychronous training like synthetic gradients).
> I'm not sure what evidence the performance of these systems show us, as bees are pretty smart for their size.
What do bees do that is remotely as demanding as being able to translate between English, French, German, Japanese etc at very high quality?
>What do bees do that is remotely as demanding as being able to translate between English, French, German, Japanese etc at very high quality?
This seems more impressive than anything any one NN can do, but less impressive than what all NNs can do: https://en.wikipedia.org/wiki/Bee_learning_and_communication
[1]: http://timdettmers.com/2015/07/27/brain-vs-deep-learning-sin...
You obviously cannot reach human-level performance on ImageNet or do anything at all with a single neuron, so such an absurd equation refutes any such calculation or attempt to use a comparison as an upper bound.
I also think neural networks could save alot of parameters if they use sparse connections. That just doesn't happen to work well with our current hardware.
Transfer learning demonstrates that the 'specialization' objection is moot. The CNNs are learning large parts of what goes into general-purpose vision, not super-narrow specific things, otherwise it would be impossible to take a classification CNN and reuse only a tweaked version for image captioning, localization, tagging, segmentation etc and find the intermediate layers learning semantically meaningful things.
One thing that is for sure though is that once we have artificial general intelligence, everything changes infinitely forever.
"Hopefully machine learning techniques give us some sort of building blocks for a general intelligence."
"Hopefully knowledge graphs give us some sort of building blocks for a general intelligence."
"Hopefully Bayesian models give us some sort of building blocks for a general intelligence."
"Hopefully rule-based systems give us some sort of building blocks for a general intelligence."
"Hopefully LISP gives us some sort of building blocks for a general intelligence."
"Hopefully computers give us some sort of building blocks for a general intelligence."
People have said approximations to all of these over time. All of these probably are building blocks. But there's no reason to believe that we're about to build the top of the tower.
I believe we should understand AI in terms of what it can do for us now, and that AGI keeps appearing to be 50 years away because it's actually centuries away and we don't even understand what we don't understand yet.
I really don't see why DNN change the game fundamentally. I mean graphical models, they really changed the way of thinking in the field. DNNs in contrast are really still fundamentally building on things built by Yann LeCunn and gang.
How do you think it will change everything infinitely forever? Are you confident this would be a positive change for us and all species on earth? This is where I'm a little lost about the enthusiasm for AGI but I'm curious what others think?
The first few improvements may be tough for it, but when you consider that it could think 1000s of times faster than humans, you can imagine how this could happen very quickly.
And then once you assume that it can recursively improve itself, we enter the realm of "the intelligence with the IQ of 5000."
The person with the highest IQ ever was 190 [1]. Imagine the kind of things we could do with an intelligence of 5000. We're talking atomic reassemblers, grand unified theories, and (my dream) a personal paradise Matrix for everyone [2].
[1] https://en.wikipedia.org/wiki/Marilyn_vos_Savant [2] http://www.smbc-comics.com/?id=3183
Note, this sidesteps all ethical alignment issues. This is assuming that the intelligence is on the same team as us. My thought process is "we're all going to die anyway so let's give it a shot," but a lot of people disagree with me (which is fair).
I think AI will split into two sets, biological inspired that will be general purpose. The other will be tool, much moreasier alien and specific to a domain. Bit like how aeroplanes don't look like birds.
Deep neural networks are great at recognizing patterns, but they still don't give us much in the way of reasoning. So, "I should start cleaning now because lunch is at 1:00 and it takes uncle Bob 30 minutes to drive over here..." kind of stuff is - so far - pretty much outside of the realm of ANN's.
If / when we have systems that can utilize the best of probabilistic pattern matching and formal logic, I think we'll see a dramatic improvement in the "intelligence" available.
Note that there is no reason, in principle (at least that I'm aware of) that "formal logic" might not ultimately turn out to be an emergent aspect of a sufficiently complex ANN. Given that our brain can do formal logic, and does at times, and is "just" neural networks as far as we know, it seems likely. That said, in the short-term, I'm guessing we'll have hybrid systems that explicitly feature various AI techniques that have been developed over the years, sharing some representations (of concepts, knowledge, etc.). Maybe the "thought vectors" that Geoffrey Hinton has spoken of could be a step down that path. I'm also curious to see if something like rule induction using CN2 might not play a role. Or maybe it'll involve "semantic networks", something like ConceptNet. Who knows?
If I had to summarize, I'd say the answer is something like "coming up with some form of knowledge representation that allows us to share 'thoughts' or 'knowledge' between disparate AI/ML systems".
In principle, one reason against the ANN purist's approach for replicating human intelligence is that it ignores all the things our brains already have programmed from the moment we step into the world; we obviously aren't anything close to a blank slate. So I think there's a burden on advocates of pure ANN to explain why our core intelligence or "formal logic" is in fact independent of this pre-programming.
That's a fair question, and I take no position on it. My take on ANN's is that they are useful tools. I'm not particularly interested in the degree to which they accurately replicate how the human mind works. I assume, perhaps too quickly, that the analogy is pretty close, but I admit that I can't prove that.
In either case, I still feel relatively confident that formal logic could emerge on top of a sufficiently complex ANN. One thing I don't know off hand, which would be interesting, is whether or not any specific research has bee done on proving or disproving that assertion.
It's closer to a gut feeling than anything. I believe if fairly strongly, but I haven't done any research in terms of trying to prove the point one way or the other. Don't read too much into it... it's just something I think. I may well be wrong.
I believe that this statement dismisses the fact that humans have plenty of time to train their brains before the birth based on multiple sensory inputs.
I get the impression from the 101 child development textbooks that infants display an impressive number of schemas and knowledge despite little, if none environmental exposure, but I'm not familiar with arguments linking these schemas and knowledge to exposure in the fetal stage.
Babies are exposed to the environment to some degree (noise, gravity etc) and it would be very dismissive to ignore it.
> As far as algorithmic ingenuity goes, this is pretty much all there is to it. With all the hype surrounding AlphaGo's victory this year, its success is just as much if not more attributable to data, compute power and infrastructure advancements than algorithmic wizardry.
... Like is anything about this sentence even approximately true??? There was nothing new about the dataset used (and to this day I can't understand why they use the exact datasets they do; KGS for the predictive net, and Tygen for rollout softmax - both amateur player databases, and don't even use the GoGoD database of pro players; seems other teams have comparable or better results at prediction with it), nor in using a sizable cluster to execute a go playing algorithm (limits to the size of it were and are when the algorithm one uses hits steep diminishing returns but anyhow, already MoGo was using rather big ones), nor in training on such a dataset for go playing. The only damn difference were PRECISELY the algorithms used, exactly in contradiction to the claim here! The nugget of truth there is that the algorithms aren't terribly innovative , just their application in this problem - but its training setup and targets are however literally groundbreaking.
Rewind the time a little bit, to the end of 2014, and you'll see the result of an Oxford team, as well from Google's team, that demonstrate a large improvement in the accuracy of the move predictor task - given a board position, predict the next move from an actual game, by using a convnet to do it. That was a first sign that deep learning had potential in go, though at that point it didn't make for a particularly strong player. Systems were getting to a point where they could bias the search effectively with such a convnet for decent gains when the AlphaGo result was announced, that dwarfed these already exciting advances!
So clearly its not about "just" using more computers nor bigger datasets, nor just doing the straightforward thing in applying deep learning to the problem; all of the above was done before AlphaGo, yeah it helped but there was just no contest between the 7d KGS amateur rank the best of the rest were getting and at or beyond top humans AlphaGo did.
AlphaGo came up with the second component, and actually solved a problem thought unfeasable in the computer go world - creating an evaluation function for go. The entire monte carlo tree search revolution of the mid-to-late 00' in computer go was how to sidestep the problem of evaluating if a particular board position is good or bad, by just running full stochastic playthroughs of the game and scoring them instead. AlphaGo on the other hand first created a decent-ish player network (though honestly nothing special - 5d KGS - that's the one that's finetuned by reinforcement learning and notably isn't even a part of the final configuration but just generates a large dataset effectively, cuz humans just haven't played enough games in history for this training setup), then generated a large dataset of games this network was made to play, and then trained (supervised!) a net on predicting the game outcome, given a board position (and taking just one position from each game, somewhat conservatively avoiding overfitting this way).
THIS is the genious of the AlphaGo algorithm; it is a monte carlo tree search algorithm, biased by a convnet, rolled out by a softmax, and crucially with an evaluation function that is mixed, 50%-50% with rollout scores
THAT is a completely novel algorithm, nothing in the literature is particularly like it, in particular the evaluation function and its mix with rollouts was not considered possible! And it works orders of magnitude better than any other monte carlo tree search tried since 2006, as well as orders of magnitude better than other deep learning biased approaches tried since 2014.
And to think that the least of these things; ie the from all the work done since 2014 on AlphaGo, 3 days (!!!) spend on finetuning(!!!) the prediction net by reinforcement learning to make it stronger (though still just a 5d amateur) so as to generate the needed learningset for another component is the only thing mentioned about their approach !?