Jeff Dean’s Lecture for YC AI [video]
blog.ycombinator.com
blog.ycombinator.com
Where they used one model to do image recognition, speech recognition and translation in the same network.
Personally I didn't find it very satisfying, I imagine something more fundamental and self referential.
His slide on "learning to learn" has a goal of removing the ML expert in the equation. Can somebody who's more of an expert in the field comment on how plausible it is? Specifically, in the near future, will we only need ML people who do research, due to the application being so trivial to do once automated?
Maybe that currently is some parts of the job of an ML engineer. But if that's the only part, I don't think that role should be called one of ML engineer anymore
ML works very well in bounded/closed domains like image and sound recognition. Open-domains are much more challenging.
Building predictive models from data in specialized domains often require insight, which machines cannot provide. For instance, let's say you collect a bunch of data and are trying to predict sales. You need to apply domain knowledge, experience and intuition to know what variables are causal or correlative. If you just throw all the variables into the mix and build a model from that, you will end up with a model that overfits badly.
There are automated "variable selection" techniques that can help to prevent overfit, but they are mostly imperfect because machines can only detect correlation and not causation. Also, many regression/classification techniques are easily fooled by noise and highly nonlinear relationships. We did some work a few years ago comparing predictive models built from a ton of sensor data (with automated variable selection) vs. one that was parsimonious that was built on select data that we knew accounted for 80% of the effect. The latter model was far superior. Noise/non-causal variables often don't just "wash out" even with very good variable selection algorithms.
It takes domain knowledge to figure out what variables matter and what variables don't.
Why?
Alternatives exist (PyTorch by facebook, theano, caffe, etc.) in any case, and the best solution wins. (in the same way scikit-learn once eclipsed all other ML libraries, but some would still use xgboost which had superior Gradient Boosting Trees).
Minor quibble, but: not really. The earliest solution wins, e.g. rails. It has to be good enough, but after a certain point "best" stops mattering.
Minor minor quibble, but: not really. The earliest solution wins, unless there is a 10x better solution, e.g Google. There were a lot of good enough options, but after a certain benchmark "earliest" stops mattering.
When you see a useful program that helps you be much more productive, you don't really care if it was coded in c++ or python.
The one part Google does care about is being able to sell Cloud TPU, but I don't think that by using this tool you're instantly in under their control.
Would it be bad if TensorFlow became the default library for deep learning? For companies that aren't Google, yes. Many organizations don't like how Google's controlled Android development.
But I think an important part of Dean's lecture are the slides that talk about making ML/DL expertise obsolete. That's the future, whether you use TensorFlow or not.
To give you an idea of how much compute it took, they spent two million dollars for one run on a relatively toy dataset, CIFAR-10. Imagine how much it would take on an imagenet sized dataset! Can your company afford ~$20 million+ per dataset?
I do think that hyperparameter twiddling aspect might get automated, but believe me, that is much welcomed by the DL research community! I would much rather spend my time on new ideas rather than trying ten different initializations :)
Chromium is very open. In many cases, Google has to ask permission from outside contributors to commit code, since the outside contributors are the "boss" of part of the codebase.
Tensorflow is closer to chromiums model. There are a lot of external contributors.
Could you maybe comment on why this is the case? Are there technical reasons for that?
(Warning: I'm biased. I compete VERY heavily with anything google including cloud and their deep learning framework.)
I would be careful about having only 1 company's use cases served. Competition forces parties to innovate.
For example mxnet and cntk or even more recently caffe2 serve different use cases and companies.
Giving you a recent one with our deep learning framework (I compete with all the above frameworks): We compete in the big data niche and recently found a few bottlenecks we missed because people run comparisons.
I'm completely blown away that Google was working on full-scale physical architectures that were optimized for these problems. Talk about being two steps ahead of the game!
Sadly there's still a lot of smart people without access to cash (academics), and piles of cash without smart people to get the value out of it.
The trolley problem: https://www.youtube.com/watch?v=21EiKfQYZXc&feature=youtu.be...
Perhaps one day, ML programs running eye diagnostics will be as cheap and disposable to use as pregnancy stick tests: "Point and diagnose."
EDIT, for context: https://en.wikipedia.org/wiki/Clinical_decision_support_syst...
I got that joke from a radiologist.
If you asked a similar question about who would take the blame if a piece of X-ray equipment was not working properly and caused some form of harm, the liability might fall on the X-ray machine manufacturer who guaranteed a certain level of accuracy, or it might fall on the hospital that procured the (potentially unsafe/certified) equipment. It might also fall on the physician who wasn't following the proper protocol for using the equipment.
Feel free to reach out to me if you are curious. I can talk in great detail.
BTW, you can purchase insurance for activities like these, though I cannot say I've seen a legal case yet, so I dont know exactly how it would work out.
I don't buy the argument that it is the same as any other equipment, although I can see companies trying to push the liability on the healthcare provider as the one making the final call.
I think the situations would be viewed identically. I don't know the exact processes, but lots of documentation would go into it and if the lab that gave the errant lab test has a history of poor results, they may get investigated and closed down.
If this was a one-off error, that will get chalked up to a one-off error. Even if the patient dies.
If that turns out to be wrong, the programmer will just point to the other 99 people successfully treated and shrug.