AI, Deep Learning, and Machine Learning: A Primer [video]
a16z.com
a16z.com
Those who ignore history are doomed to repeat it. As Roger Schank pointed out recently http://www.rogerschank.com/fraudulent-claims-made-by-IBM-abo..., another AI winter is coming soon! Funny that the video details the three first AI winters but the author doesn't realize that this excessive enthusiasm in one particular technique is contributing to a new one!
Especially dangerous is going to be the mix of machine learning with healthcare. I believe Theranos tried it and found out it's not that easy... I'd watch this space with skepticism.
Medical diagnosis has been one of the primary application areas of AI since the 70s (maybe earlier, I can't remember off the top of my head). The widespread non-availability of automatic doctors should tell you how well that has worked :-(.
Coincidentally enough I worked both in AI (1980s) and drug development (2000s) and now really understand how hard it is!
I do believe we will soon see automated radiology analysis as it is likely to appear to be most amenable to automated analysis. Presumably in Asia first as the US FDA will justifiably require a lot of validation. The opportunity for silent defect is quite high -- you are right to say "especially dangerous"
The value of AI depends largely on perceived value IMHO, and the frequency of "winters" will correlate with that. I think we are still a bit too early for VR to really take off, but that did not stop a $2b acquisition and loads of investor interest. This will probably artificially constrain what should be an AI winter right now, just because so much money is continuing to go into it.
I personally applaud that so much enthusiasm is going into AI right now, and though we are repeating history to some extent I still think we are making incremental advancements (however small) - even if this just means applying old AI techniques to new advancements in hardware.
The video at least mentions Yann LeCun's early work with convnets, but there's no mention of Hinton et al's work with RBMs and DBNs in 2006, or Bengio et al's work with autoencoders in 2006/2007, or Schmidhuber et al's invention of LSTM cells in 1997... I could keep going. The list of people whose work is insulted by omission is HUGE.
I stopped watching the video at that point.
He gives a "brief history of backpropagation" here: https://youtu.be/l2dVjADTEDU?t=4m35s
Throwing a phenomenal amount of neurons at a problem is not a goal; using a minimal amount to solve it in a given time budget is.
The statement at the end of the video, “all the serious applications from here on out need to have deep learning and AI inside”, seems awfully misguided. Even DeepMind doesn't use deep learning for everything.
I found these helpful while researching the history: http://www.andreykurenkov.com/writing/a-brief-history-of-neu... http://www.scholarpedia.org/article/Deep_Learning
What else have you found particularly useful?
http://www.iro.umontreal.ca/~bengioy/papers/ftml.pdf
http://people.idsia.ch/~juergen/deep-learning-conspiracy.htm...
Were I in your shoes, I would NOT have highlighted the Google YouTube experiment as "the" big breakthrough. It was just an interesting worthwhile experiment by one of many groups of talented AI researchers who have made slow progress over decades of hard work. Why single it out?
--
PS. The YouTube experiment did not produce new theory, and from a practical standpoint, it would be unfair to say that it reignited interest in deep learning. Consider that the paper currently has only ~800 citations, according to Google Scholar.[1] For comparison, Krizhevsky et al's paper describing the deep net that won Imagenet (trained on one computer with one GPU) has over 5000 citations.[2] And neither of these experiments deserves to be called "the" big breakthrough.
[1] https://scholar.google.com/citations?view_op=view_citation&h...
[2] https://scholar.google.com/citations?view_op=view_citation&h...
The reigniting of deep learning around 2012 was because of Krizhevsky, Sutskever & Hinton winning the Imagenet challenge (1000 object classes)
Contrary to how much Google tried to sell Andrew Ng's "breakthrough" 2012 experiment with tons of PR, the paper is very weak, and cant be reproduced unless you do a healthy amount of hand-waving. For example, to get an unsupervised cat, you have to initialize your image close to a cat and do gradient descent wrt the input. Or else, you dont get a cat... It is not even considered a good paper, forget being breakthrough. Also, those 16000 CPU cores etc. can be reproduced with a few 2012-class GPUs and much smaller time-span than their training time.
The next slide after the 2012 breakthrough that shows the Javascript neural network -- contrary to what it looks like -- is not TensorFlow either. It has cleverly and conveniently been given TensorFlow branding, so most people just confuse it, but it's just a separate Javascript library akin to convnet.js
Since your page gets a ton of hits, it's at least worth it to publish a comment about these GLARING inaccuracies.
1. They demonstrated a way to detect high level features with unsupervised learning, for the first time. That was the main stated goal of the paper, and they achieved it magnificently.
2. They devised a new type of an autoencoder, which achieved significantly higher accuracy than other methods.
3. They improved the state of the art for the 22k ImageNet classification by 70% (compare to 15% improvement for 1k ImageNet in the Krizhevsky's paper).
4. They managed to scale their model 100 times compared to the largest model of the time - not a trivial task.
You say "it can't be reproduced" and then "can be reproduced", in the same paragraph! :-)
Regarding initializing an input image close to a cat to "get a cat", I think you missed the point of that step - it was just an additional way to verify that the neuron is really detecting a cat. That step was completely optional. The main way to verify their achievement was the histogram showing how the neuron reacts to images with cats in them, and how it reacts to all other images. That histogram is the heart of the paper, not the artificially constructed image of a cat face.
It's not perfect, but I cant give a reasonable answer to the other extreme of an opinion. fwiw, as a researcher I spent quite some time on this paper, but that subjective point doesn't mean anything to you.
The only negative thing I can say about that paper is they have not open-sourced their code.
Not to mention the use of ReLu and understanding of the Vanishing Gradient issue
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&e...
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&e...
Saira Mian, Micheal I. Jordan (Andrew Ng was a pupil of his) and David Blei were not mentioned in this video so they are off the mark a bit. Vector space is the place.
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&e...
AI has become the most competitive academic and industry sector I've seen. Firms like Andreessen are trying to understand the impact during this AI summer and they should be applauded for this.
One of the keys to AI is found here: https://www.google.com/webhp?sourceid=chrome-instant&ion=1&e...
Deep learning has very little to do with how the brain and mind work together. In the video the highlight on ensemble (combinatorial) techniques are a big part of the solution.
Most deep learning people I talk to acknowledge the algorithms & data structures are only lightly inspired by brain anatomy. Michael Jordan makes this point well in this IEEE article: http://spectrum.ieee.org/robotics/artificial-intelligence/ma...
Donna Dubinsky and Jeff Hawkins and the team at Numenta are doing the most explicit biomimicry work I'm aware of. What else is happening that you know of?
Stephen Pinker gave a somewhat unbiased analysis of CTM (Computational Theory of the Mind) a while ago. Very relevant.
Also, thanks for that term! Was not aware of it, but very useful to describe what I think is one of the big problems.
Care to elaborate? Certainly classical computers don't look a whole lot like the information processing of any living being (I guess one could argue that they mimic what humans do with pencil and paper, but it seems a bit of a stretch.) To me other things like say human-made engines also don't look that similar to any life form, but then I think I know very little outside computing, relatively to say the average person here, and hence I'm genuinely curios about your remark and "wouldn't be surprised to hear something surprising."
e.g. Austria - Capital = Vienna OR 12 x 12 = 144
We continue to mimic nature in our scientific endeavors and the brain, as the best pattern matcher we know of, is no exception.
I see increasing compute power, an increased learning set (the internet, etc), and increasingly refined algorithms all pouring into making the stuff we had decades ago more accurate and faster. But we still have nothing at all like human intelligence. We can solve little sub-problems pretty well though.
I theorize that we are solving problems slightly the wrong way. For example, we often focus on totally abstract input like a set of pixels, but in reality our brains have a more gestalt / semantic approach that handles higher-level concepts rather than series of very small inputs (although we do preprocess those inputs, i.e. rays of light, to produce higher level concepts). In other words, we try to map input to output at too granular of a level.
I wonder though if there will be a radical rethinking of AI algorithms at some point? I tend to always be of the view that "X is a solved problem / no room for improvement in X" is BS, no matter how many people have refined a field over any period of time. That might be "naive" with regards to AI, but history has often shown that impossible is not a fact, just a challenge. :)
Machine learning and human learning happen in much the same way. We have a dataset of memories, and we have a training dataset of results. We then classify things based on pattern matching. The current human advantage is an ability to store, acquire and access certain kinds of data more efficiently, which helps in solving a wider variety of problems. For problems in which machines have found out how to store, acquire and access data more efficiently (such as chess) machines are far superior to humans.
Right now the focus is on solving important specific sub-problems better humans, rather than the ability to solve a much wider variety of sub-problems.
The focus is there because there are business applications and money there. Do researchers really think that some version of a chess-bot or go-bot or cat-image-bot or jeopardy-bot will just "wake up" one day when it reaches some threshold? That this approach is truly the best path to AGI?
A machine can play chess better than a human because the human used its knowledge to build a chess-playing machine. That's all it can do. It takes chess inputs and produces chess outputs. It doesn't know why it's playing chess. It didn't choose to learn chess because it seemed interesting or valuable. No machine has ever displayed any form of "agency." A chatbot that learns from a corpus of text and rehashes it to produce "realistic" text doesn't count either.
You could argue many of the same things about humans themselves. Consciousness is an illusion, we don't have true agency either, we also just rehash words we've heard - I believe these things. But it seems clear to me that what is going on inside a human brain is so far beyond what we have gotten machines to do. And a lot of that has to do with the fact that we underwent a developmental process of billions of years, being molded and built up for the specific purpose of surviving in our environment. Computers have none of that. We built a toy that can do some tricks. Compared to the absolute insanity of biological life, it's a joke. I think it is such hubris to say that we're anywhere close to figuring out how to make something that rivals our own intelligence, which itself is well beyond our comprehension.
On top of that, once balance is learned it's instantly transferable to other "machines" where as each human has to learn it.
2) are computers that can understand speech, recognize faces, drive cars, beat humans at Jeopardy really 'nothing at all like human intelligence?'
The moving goalpost has been less an issue of "what is AI" and more an issue of "what are difficult task at the edge of AI research". People with passing interest (even most programmers) don't distinguish between the two. Of course "difficult tasks in AI research" is a moving goalpost and it will be moving until we achieve general intelligence and beyond. This is a requirement for progress in AI research. If those goalposts stop moving before we have a general intelligence then something is wrong in the field.
When researchers (not the general public) start arguing whether the goalposts should be general intelligence or super intelligence that is when we know we have traditional AI. When we try to figure out how to get adult human level intelligence to take hours or days to train on the top supercomputers rather than months or years -- that is when we have AI. Even then, if the training part requires that much computational intensity, how many top supercomputers are going to be dedicated to having a single human level intelligence?
You could train current algorithms used in AI research for decades and have nothing resembling general human intelligence.
Only at such a high level of abstraction as to be meaningless.
> 2) are computers that can understand speech, recognize faces, drive cars, beat humans at Jeopardy really 'nothing at all like human intelligence?'
They are not. Hundreds of man years worth of engineering time go into each of those systems, and none of those systems generalizes to anything other than the task it was created for. That's nothing like human intelligence.
I'm not sure what this means or how the abstractions are meaningless? From Gabor filters to concepts like "dog", the abstractions are quite meaningful (in that they function well), even if not to us.
> They are not. Hundreds of man years worth of engineering time go into each of those systems, and none of those systems generalizes to anything other than the task it was created for. That's nothing like human intelligence.
This isn't strictly true if we look at the ability to generalize as a sliding scale. The level of generalization has actually increased significantly from expert systems to machine learning to deep learning. We have not reached human levels of generalization but we are approaching.
Consider that DL can identify objects, people, animals in unique photos never seen before and that more generally the success of modern machine learning is it's ability generalize from training to test time rather than hand engineering for each new case. Newer work is even able to learn from just a few examples[0] and then generalize beyond that. Or the Atari work from DeepMind that can generalize to dozens/hundreds of games. None of those networks are created specifically for Break Out or Pong.
It's also not entirely fair to discount the hundreds of years of engineering considering most of these systems are trained from scratch (randomness). Humans, however, benefit from the preceding evolution which has a time scale that far exceeds any human engineering effort. :)
Computer doesn't need to be strong AI to replace human.
And most of them don't transfer.
What matters here is the concept itself (deep learning as a generic technique) but also scalability. Not the specifics that we have today, but the specifics that we will have 20 years from now.
The concept is proven, all that matters now is time...
This is a very naive point of view. You could deep-learn with a billion times more processing power and a billion times more data for 20 years and it would not produce a general artificial intelligence. Deep learning is a set of neural network tweaks that is insufficient to produce AGI. Within 20 years we may have enough additional tweaks to make an AGI, but I doubt that algorithm will look anything like the deep learning we have today.
2) These things are all very human-like, but they are still sub-problems IMHO :)
Hopfield nets for example provide associative memory.
It may not all be groundbreakingly efficient, but very worthwhile.
Then immediately he continues that
"So it turns out that using deep learning techniques we've already gotten to better than human performance [....] at these highly complex tasks that used to take highly, highly trained individuals... These are perfect examples of how deep learning can get to better than human performance... 'cause they're just taking data and they're making categories.."
I think that brushing off the dramatic social changes that this technology will catalyze is irresponsible.
One application developed by one startup in California (or wherever) could make tens of millions of people redundant all over the world overnight.
How will deep learning apps affect the healthcare systems all over the world? What about IT, design, music, financial, transportation, postal services... nearly every field will be affected by it.
Who should the affected people turn to ? Their respective states ? The politicians ? Take up arms and make a revolution ?
My point is that technologists should be ready to answer these questions.
We can't just outsource these problems to other layers of society - after all, they're one step behind the innovation and the consequence of technology is only visible after it's already deeply rooted in our daily habits.
We should become more involved in the political process all over the world (!) - at least with some practical advice to how the lawmakers should adapt the laws of their countries to avoid economic or social disturbances due to the introduction of a certain global AI system.
we (humans) want ai and will all be better off for it. we are heading into a shift on the order of the industrial revolution and the best course is unknowable. we should harness this technology, try to distribute it bu collaborative building & information dissemination and study the idea of personal fullfillment & a basic quality of life.
In short, the answer is definitely not to stop working on it because it is just basic game theory that someone/entity will not. We need to leverage to fix the problem it creates & learn how to allocate our resources by studying this immeadiately.
This is true for some tasks - but not true for others, even if AI could in theory do them better.
For example, a medical doctor. It is easy to see this as just a job, but it's so much more than that - it's the culmination of 20 years of studies, a childhood dream, the feeling of being important and useful to society, the "thank you" from the patient, the social circle, the social status...
There are many people who actually enjoy their jobs, because it gives them meaning and satisfaction.
Another example - musicians. I see so many "AI composer" projects out there - algorithms which compose music... I think these people are kind of missing the point ..
It's easy to see music as notes and tempo, but it is much more than that. It is a medium, a tool, through which the listener can connect to the artist and experience his emotional state: https://www.youtube.com/watch?v=1kPXw6YaCEY
Having an algorithm on the other side feels so fake.. artificial..
So what is the right course according to you? Do we set an artificial threshold at some point in the future, where these jobs will be swapped to AI? Do we simply never replace them with AI?
Do we slow progress in order to cater to people's obsolete expectations (don't want to hurt their feelings after all).
No. Problems will appear and we will solve them. It is our duty to the universe, to ride the wave and see it all the way through.
EDIT: Virtual Reality will solve a lot of the "meaning of life/emotional satisfaction" issues that creep up. And VR is getting a jump .... right about now. It's quite amusing, just thinking about the timing.
I've done my fair share of acid and mushroom trips, explored the DMT hyperspace dozens of times, travelled through time and space into alternate realities, met and communicated with biological and machine beings and have thoroughly explored our planet.
My mind is wide open :)
Since you mentioned the Universe, then it would help to remember that in "galactic" terms, we've just climbed down from the trees a couple of hundred thousand years ago and our technology is still extremely primitive and it is not at all a proven fact that we can survive it.
More down to Earth, what I suggested in the parent post is that people who are bringing this technology forward should also be the people who come up with the political and social proposals for changes necessary to accommodate it.
We can't just unleash these "technological atomic bombs", which fundamentally change the social game and then expect the politicians to handle the fallout. Tech people and scientists need not stay on the sidelines any more - they must be the designers not only of technology, but of the social system too, since the two are merging anyway...
Of course this can't be "enforced", rather, it's an ethical thing to do on the part of tech companies, justified by the fact that deep down, our motivation is to make the world "better".
My point is that the process, if we might call it that, will be osmotic and not discrete. Of course there is also the notion of people believing that there is a layer of central planning somewhere and that "we" (groups, organizations, nation states) actually have high-level organizational control over what is happening.. Watching Taleb talk about tail events will quickly put these notions to the ground.
We shouldn't waste time inventing such control when there is none or even inventing the illusion that such control will be effective. All we can do, I feel, is guide.
In fact something as simple as naive Bayes will work reasonably well for that.
I'm not sure if you are aware, but in (say) image classification it's pretty common to take a pre-trained net, lock the values except for the last layer, and then retrain that for a new classification task. You can even drop the last layer entirely and use a SVM and get entirely adequate results in many domains.
Here's an example using VGG and Keras: http://blog.keras.io/building-powerful-image-classification-...
And here a similar thing for Inception and TensorFlow: https://www.tensorflow.org/versions/r0.8/how_tos/image_retra...
However: Zero-Shot Learning Through Cross-Modal Transfer http://arxiv.org/abs/1301.3666
One main thing it lacks is imagination. Humans can learn things and can imagine different combinations of those things. For example, if I ask you to imagine a Guitar playing Dolphin, you could imagine it and even recognize it from a cartoon even though you have never seen it in your life before. Not so for Deep Learning, unless you provide massive amount of images of Dolphins playing guitars.
Perception works at 10-20 frames per second, all day long. That means 0.5 million perceptions per day. Why would a small neural net that has 1/1000 the experience of a child and 1/1000 the size of a brain (assuming it is 100 million neurons which is huge!) be able to be more accurate?
What you are referring to is called "one shot learning" and is the ability to learn from a single example, which is being studied in the literature.
These days one should be careful when claiming things deep learning can't do. There are in fact systems that can imagine things it has never seen before.
Here's an example. It takes any text, and imagines an image for it: https://github.com/emansim/text2image
Here's a more recent example, with even higher quality images: http://arxiv.org/pdf/1605.05396v2.pdf
On a purely visual level - let's say we have 10k of static 32x32 images defining a class of cat. Or even more of them plus some negative examples. Each image is a different cat, in a different position (they're incredibly flexible creatures). Having so many cases we should be trying to make some kind of a generalization of what 1024 pixels of a cat should look like.
A family with a toddler has only one cat. From that one example, he learns the concept of what a cat is and is able to generalize it when in different situations. But a toddler has 2 eyes, his visual input is stereo. Even if he sees just one cat, it's not a static image interaction. The input is temporal, he can see the cat moving and interacting with the environment.
This is typically referred to as "one-shot learning" in the literature -- and people certainly work on it!
On a side note I always admire the polish of the content that comes out of a16z - its typically very well put together.
Apparently, robots still struggle to pick up things?