Machine Learning Crash Course
developers.google.com
developers.google.com
Yes, AI/ML MOOCs teach the corresponding tools well, and the creation of new tools like Keras make the field much more accessible. The obsolete gatekeeping by the AI/ML elites who say "you can't use AI/ML unless you have a PhD/5 years research experience" is one of the things I really hate about the industry.
However, contrary to the thought pieces that tend to pop up, taking and passing a crash course doesn't mean you'll be an expert in the field (and this applies for most MOOCs, honestly). They're very good for learning an overview of the technology, but nothing beats applying the tools on a real-world, noisy dataset, and solving the inevitable little problems that crop up during the process.
Reviewing the Keras documentation (https://keras.io) and examples (https://github.com/keras-team/keras/tree/master/examples) are honestly much better teachers of AI/ML than any MOOC, in my opinion.
(Of course, Keras is now a part of TensorFlow, so there's a neat Google vertical intergration with this crash course!)
This is not true since a few years ago. But the fact that you can use it doesn't mean you understand what is happening and why it works in development but not in production. Everybody can copy a jupyter notebook and train a TensorFlow model in ImageNet. Now go to a new domain with very few information like 3D models and create a new network to be trained in that dataset. How many people that can train ImageNet can do the latter? Even inside deep learning experts in image classification fail in reinforcement learning domains and need a couple of years to be completely productive.
So what are they hoping to achieve with this course? I'm genuinely asking because part of me wants to take the course, but another part of me feels like what's the point if, even through many additional courses to build up a skill set, Google wouldn't hire you as an ML engineer unless you basically start your career back to a junior engineer but in machine learning at another company.
If the answer is that you're waiting for a special sign that it's worth doing before making an effort, then that really tells me that your enthusiasm for doing ML is not reality-based. Doing the ML thing is a pretty different mindset from other software jobs.
I agree with your general thrust, and you're right, messy data is often 95% of the problem, but even going through just the Google courses will put people in the top 15% in most cities.
Edit: To add my perspective, with years of industry experience and graduate-level machine learning coursework, I have never before encountered this term.
[0]: https://link.springer.com/chapter/10.1007/978-3-319-16354-3_...
I find people in Math and CS have often very different names for the same type of concepts and they could easy understand each other if they stuck to the more common terms.
In this case, saying: TYPE 2 ERROR, makes you look like you are trying too hard.
that said, if you are just pointing to a box in a confusion matrix and saying "TYPE II ERROR," you are probably trying too hard.
[0] https://en.wikipedia.org/wiki/Type_I_and_type_II_errors#Tabl...
...and if you haven't come across that either, see https://en.wikipedia.org/wiki/Jaccard_index for details.
I had to google it. It's a false negative.
A "Type 1" error, is a false positive.
Is this like how people overuse the term "orthogonal"?
A false positive or false negative, can be like a pregnancy test.
A false positive, can be where the pregnancy test shows your wife is pregnant, but she is not. And the baby never arrives. Phew, dodged a bullet!
A false negative, can be where the pregnancy test shows your wife is not pregnant, but she really is. And 9 months later, a baby accidentally pops out. Oh crap!
Otherwise, please take this wisdom from programmers, who deal with this sort of thing all the time, and use an enumeration, in this case, {False Positive, False Negative} will do just fine.
https://en.wikipedia.org/wiki/Curse_of_knowledge
What you are describing also sounds a little like the Dunning-Kruger effect:
Funny you mention Jaccard; I was looking up if IoU (Intersection over Union) has any other name known to ML people when I was preparing my self-driving car presentation (IoU is used in semantic segmentation), and found out it is called Jaccard index as well. To my surprise, all ML experts I know knew about IoU but nobody about Jaccard. I guess it might depend on which university you attended?
It will be interesting to see how the situation evolves but my own observations are that people trying to enter the space might be better off getting a quickie masters if they can afford the time or cost than to try and bootstrap it.
Not a fan of this "data scientist is a unicorn" style of thinking. The best people in any profession (especially software engineering) also use these skills in their day-to-day work.
However personally I view the rest of the opposite way round. Getting through a course on Deep Learning takes months [0]. Then reading through Keras code once you understand the appropriate NNs is easy.
For example it takes a while of going through Neural Networks to understand ResNets. But if you understand ResNets then looking though Keras code that creates a ResNet [1] is easy.
If I want to build a NN of any sort in Keras I can just Google for it. However there's no simple Googling you can do to teach yourself NN in an easy to follow structured way.
[0]: https://www.deeplearning.ai/
[1]: https://github.com/Hyperparticle/one-pixel-attack-keras/blob...
Also, DL != ML.
Paraphrasing "The Tao of Network Protocols": If all you see is DL, you see nothing.
For many current applications of ML this is acceptable because you're just stealing an idea from a paper or stealing ImageNet to recognize your problem. For anything else you really need to pay up and fight with Google for a real expert.
Which phd, though? All PhDs are not equal (see politics vs computer vision). Also, PhDs are hardly the holy grail of demonstrating capability, accuracy or intellect, especially given the reproducibility crisis, phds as a measure of any of those things should be used carefully.
Wonder where I've heard this one before. :)
It is absolutely true that you do not need a graduate degree to apply AI/ML to vanilla problems.
It is also absolutely true, in my experience, that you need a graduate-level education or years of hands-on experience to troubleshoot cases where AI/ML fails on a deceptively-simple problem, or to tweak an AI/ML algorithm (or develop a new one) so it can solve a novel problem.
That said, I think these MOOCs are good enough to get someone to a place where they can create nice /r/dataisbeautiful-style visualizations, or pair with a senior-level DS to deliver something.
(Edited to add folks who have worked on problems for years and add a final note.)
Not to sound like I walked uphill in both directions back in my day or something, but I remember building models in numpy without pandas. It was tedious -- and that's just a nice API wrapping ndarrays!
Local minima?
You can make an argument current tools haven’t really surpassed a Lisp Machine for developer productivity, or a SmallTalk environment.
We're trying to re-write our recommender for artist music stations at iHeartRadio (aka "I'll listen to Drake or Kendrick Lamar's station at the gym today"). Just today, I tried adding negative sampling to the matrix I'm factorizing, hoping it encourages spread in the embeddings learned for artists in certain types of genres.
I have a MS, but not a lot of research experience. It would have taken me a while to find this solution on my own. However, the moment I described this problem to my manager - a PhD graduate with several years of research and industry experience - he immediately suggested negative sampling.
What I learned during my MS helped me grok the math immediately. We're adding noise to the training set and penalizing vectors lengths to avoid overfitting. Easy! Identifying a solution worth exploring? Not easy, at least without a degree or significant experience.
(There's also the chance I should know this, in which case I have some reading to do. ¯\_(ツ)_/¯)
3D programming started as a field where only PHD's had any deep understanding of what was going on simply because they had experience when nobody else did. You see this pattern repeated frequently, in any complex domain.
The PhD is sufficient but not necessary here, right? A PhD researcher's job description is basically "learn necessary math, become a domain expert, and publish papers advancing that domain." It's difficult (but possible) to gain the same experience in industry if you don't have a graduate degree. Which company would pay you to work through Bishop or Goodfellow for a few months? Even a principal DS doesn't get that deal, much less a junior/associate.
Also remember: my comment addressed non-vanilla cases. In your example, this is the difference between a researcher advancing 3D programming and someone using Unity or Unreal.
(Also, sorry for all the edits. Done now!)
Machine leaning for sorting peas at high speed is a very well trodden area at this point with a lot of industry specific domain knowledge. I expect self driving cars for example to reach a similar state in ~10-25 years.
The risk with a PHD is you miss the specific wave. But, if you want to stay on the bleeding edge it's probably well worth it.
Yep! We’ve now made our way back to my initial point in response to OP. :)
I am a good ways through my masters (second CS degree, first specializing in ML), and the more I learn, the more I realize that on any given topic, there is no guarantee the PhD in the room has the most expertise. Machine learning is a broad field that contains many subfields, methodologies, and many applications. It is a bit like computer systems or software engineering: nobody knows it all, people who are experts have intimate knowledge of a specific subset of the field. Of course, you can more around over time, but it takes years to build up expertise in even two or three subfields of machine learning.
Side note: sounds like we do similar work. I work at Vevo, also do a lot of matrix factorization to learn latent factors of items such as artists, videos, etc.
Sure thing, but someone in that position needs years of experience in recommender systems, as well as working with researchers.
Folks are hanging on to the PhD part of my claim, instead of the "PhD or experience" part. The fact is, a PhD + prior industry work means the person is close to a decade of relevant background, grad degree or not. They will unstick a co-worker far faster than an experienced backend developer with, say, a year of Keras experience.
> Side note: sounds like we do similar work. I work at Vevo, also do a lot of matrix factorization to learn latent factors of items such as artists, videos, etc.
Seems like it! Email me if you'd like to chat some more offline (it's in my profile).
If you take some sophisticated deep neural net and try to train it on a binary classification where tails occurs 99% of the time - unless you specifically take measures to correct for this bias - the net will just learn to predict tails.
How much of that is critical domain specific knowledge and how much of that is just general engineering debugging/problem solving experience though? Certainly the person who does have the masters/PhD and a few years of applying that to real-world ML problems will have the edge but an experienced developer who's got a knack for maths (though no direct ML experience) may be able to get up to speed quicker than you think. Part of that will be experience with knowing how and when to ask the right questions when you get stuck.
It's both, right? You pick up problem-solving techniques as a researcher or engineer; as the former, those techniques lean towards scientific problems. Your average engineer doesn't need to know about contrasting.
Again: it's possible to learn the necessary math in your spare time! I agree!! However, it's far easier to do it in a graduate program as a full-time job for 2-5+ years.
Gatekeepers shouldn't keep gates just for gatekeeping sake. But if so-called ML experts don't even know undergraduate computer science, that should really give you pause before you open up your wallet for them.
That's fine if you want a technician (and if they're charging technician's rates).
Considering that ML is really a CS-oriented form of statistics, why would you expect a statistician to know CS theory?
Sure, you don't need to know the halting problem to approximately solve MNIST by fitting a million-parameter curve to a dataset.
But you're misleading people if you're claiming to have any kind of insight into how computers can be made intelligent, or how computers can "learn", when you don't even know the halting problem.
And to consider these algorithms to not "learn" is similarly unfair. They do. They learn to solve specific problems (at least right now), but they do learn.
The more you look around, the more you find science concepts are named for marketing purposes.
Heck, "data scientist" is a bit of nonsense.
Turing's work on the limitations of his machine are interesting, and I'm sure people with a deep understanding of them can advance the study of computation.
I think you're just being dismissive of skillsets which aren't your own. I think you're just bothered by the fact that AI and ML are being advanced more by people with more knowledge of linear algebra and statistics than computer science. And realize that it's the arrogant among them that will dismiss you as "just a technician."
Anyone who is looking down on either "scientists" or "technicians" should get over themselves.
Not OP, but I'm working a lot with ontologies. Some ontologies representations are undecidable, while other languages are not very expressive but can be manipulated in polynomial time. Had I not known that, I would still be like "crap, why does it take so long? I must have a bug somewhere, maybe I should switch to C".
> AI and ML are being advanced more by people with more knowledge of linear algebra and statistics than computer science.
Just answered OP about that, but actually, symbolic AI is pure computer science. It does not get as much publicity as ML currently, but believe me, it's everywhere: at the core of almost all package managers, like debian's apt-get or maven, at the core of most advanced static code analyzers, etc.
I'm working in the symbolic AI field, and sometimes use ML techniques. They are complementary. To me, ML is about induction, AI is abut deduction. They don't solve the same kinds of problems and they tend to work pretty well together.
that said, i'd also expect a phd in statistics to be able to figure out boosting without taking an undergrad course that worked up from automata. so the halting problem test, while it does capture something, may not be quite right.
To be fair, Machine learning seems more closely related to applied mathematics - statistics/optimization than to computer science.
I'd suppose that most developers, formal education or not, would have encountered an infinite loop at some point in their initial work with iteration or recursion.
How does knowing that Turing proved you can't predict this bug in a piece of code change anything?
I might genuinely be missing something important here - not trying to be snarky in my questioning.
It seems like obviously infinite loops are a disastrous bug for critical code - but what does knowing the formal name of the problem and background of its discovery give you?
I could understand if you were arguing in favor of test code or static analysis.
"I attended a big software dev conference. Someone I met did not know about data bias. They heard of gradient boosting but nothing more than hearing them. If so-called dev experts don't even know undergraduate statistics, that should really give you pause before you open up your wallet for them."
The problem is that having a hammer makes one see everything as a nail. Sure, given a suitably clean set of images, anyone who's done a couple of tutorials will be able to apply a pre-trained neural on them to get something.
The hard part is getting an understanding of what tweaks to use when, and when to give up on a method. Otherwise, it is very easy to get carried away and waste time/resources.
For that, one needs to develop a good understanding of the landscape of ML algorithms, why each of them works and how they could break. That typically takes (intensive) experience or an understanding of the theory. Otherwise you'll be doing a brute-force search through a list of possible algorithms. As they say, "a few days in the lab might save a few hours in the library..."
Yes, things can get painful during hiring because the process is broken as it is, with additional complications due to not knowing how to vet for quality in a nascent field. But the "ML elite" are not morons and they don't mean to be obnoxious gatekeepers.
IMO, it's much easier to pick up the programming required for ML than the statistics. This was reflected in the classes I took as a double statistics/computer science major. Most of the people in my CS department's machine learning course were statistics students looking to go into data science, not computer programmers looking to get in on the ML trend.
Any field that you can become an expert in with a 6-week course or less is not a field that should be paying even high 5-figure salaries. Or, conversely, any field which pays 6-figure salaries is either not accessible via an MOOC, or is massively overinflated and probably in a bubble.
You're stating the painfully obvious here. I doubt anyone reading HN is under the impression that they'll be an expert after a single online course.
This is just a marketing stunt by Google to ensure their tooling is the defacto standard for AI/ML so that Google can dominate the AI/ML market they way they dominated Internet Search.
It's the main reason why I decided to present a talk at the next PyCon Italy, as a very junior data scientist, to inspire other Python developers to learn some practical machine learning. If I could do it (and use it for a work project already) many other people can do (and no, I don't even have a degree in CS, just years of work experience)
I'm going to have to ask who exactly are those AI/ML elites who say "you can't use AI/ML unless you have a PhD/5 years research experience".
If anyone is looking to get more indepth, I work on an open source python library for automated feature engineering called Featuretools https://github.com/featuretools/featuretools/. It can help when your data is more complex such as when it comprised of multiple tables.
We have several demos you can run yourself to apply it to real datasets here: https://www.featuretools.com/demos.
I expect that as companies increase their focus on finding practical applications of ML / AI, the topic will start to get more attention in these tutorials, as well as from researchers. Right now, too many people assume you already have a feature matrix, which is rarely the case when working on real world problems.
The 1st place in Kaggle's Porto Seguro competition trained an Autoencoder on raw data to extract features.
Given that entire scientific careers, books, and conferences are built around the topic of feature engineering, and at least IMO good ML tools live or die with good feature engineering (in its broadest sense, for you deep learning fanatics :-)) that doesn't seem like more than the bare minimum I'd expect from any ML "crash-course" that is to be taken serious (and I wouldn't expect an ounce less from Google... :-)).
Am I missing something, maybe?
In any case, nice work of your own, and thanks for sharing it!
They understand the value, but but if you keep watching, they don’t seen go beyond the basic.
Is anyone here doing Andrew Ng's Machine Learning course [1]? I'm about half-way through and really enjoying it. I'm particularly appreciating that the programming exercises are done in MatLab/Octave, so I feel that I'm really understanding the fundamentals without an API getting in the way, and developing some good intuition. Obviously frameworks are the way to go for production ML work, but I wonder whether ML people here think this bottom-up approach is advisable or could it be misleading when I move on to Keras/TensorFlow/whatever?
[1] https://www.coursera.org/learn/machine-learning
Edit: brevity
I always advocate learning the fundamentals. Machine learning is math, and neural networks in particular rely on linear algebra and vector calculus. (You can build a NN without using linear algebra directly, likely it'll be slower and besides, the concept still relies on linear algebra).
Frameworks abstract away a lot of the mathiness, which is a net good for society (ie, exposing lots of developers to neural networks), but I consider that a net-negative for the individual developer.
When working on anything but trivial toy problems, you should make sure you understand your problem domain and implementation thoroughly. Is the activation function you've chosen ideal for your problem domain? If not, choose a better one. If no better one exists, you can invent it; but you'll also need to know how to design the backpropagation algorithm for that new activation function (which requires some vector calculus).
Learning the math, as you have, helps you tune your algorithm based on actual knowledge rather than guesswork. I don't think it will be misleading when you move on to a framework. The frameworks are built on the same math.
That said -- if all you're looking to do is play around, then you don't need the math as much.
In the past, like when I started on ML, the best tip was to make sure to do some edge detection with a few convolutions before feeding an image to a neural network. Now, we have convolutional neural networks that kinda do that for you automatically.
Sometime in between those two dates, someone figured out how to get the convolutions trained via backpropagation -- and they did that by deriving the gradient of an arbitrary convolution (or more likely, looking it up). And that let us put convolutions right in the neural net and have the convolutions automatically train themselves along with the rest of the network. And we observe that the convolutions do things that we would do, like remove unnecessary detail and highlight edges or exaggerate colors.
Anyways; I believe the current state-of-the-art for generic image feature recognition is an ensemble of convolutional neural networks. I believe Google leads the pack on the commercial side so maybe look into how they do it.
If you look at capsules papers, you will realize that convnets are not very good at recognizing transformations (e.g. 3d rotations) of the same object. That's probably why so many training examples are required to make them work well.
Also, if you look at errors made by state of the art models, some of them are obvious (to a human) objects, classified as something entirely different and unrelated. Which leads me to believe that object recognition is not completely solved until a model has some kind of common sense, either build in, or acquired during training.
I see TensorFlow as the Angular of machine learning: first on the market, powerful but unwiedly. Like Angular, it will ultimately get superseded by tools with a nicer API (scikit-learn, Keras) or more versatility (PyTorch). Like Angular, it's probably not the best choice for a beginner to invest time into.
Do it. It's worth your time. Very well paced exercises, and it walks you through the flow quite nicely.
Any book suggestions to getting up to speed in this area?
[0] https://www.amazon.com/Schaums-Outline-Precalculus-3rd-Probl...
Specifically following questions are the ones I struggle with:
1. How did you figure out what features would be useful?
2. How did you figure out what algorithm(s) are appropriate?
3. how and why did you massage the data in a specific way?
It's written for a very different set of problems than typical ML, but it has lots of really good advice for practical problems in data analysis and prediction (which is another term for ML).
Mostly people learn this stuff by experience. Find a dataset, choose a predictor, filter, clean and massage your data till you get better metrics/understanding (preferably both). Rinse, repeat on many different datasets and problems, and you'll know how to do this.
There are various feature engineering and feature extraction techniques. Filter methods, wrapper methods, and embedded methods. Principle component analysis, autoencoding, variance analysis, linear discriminant analysis, Gini index, genetic algorithms, etc -- the feature selection process will depend on the dataset, the problem domain, the analysis algorithm you ultimately use, etc.
> How did you figure out what algorithm(s) are appropriate?
Also depends on the problem domain. Discrete or continuous data? Categorical features, numeric features, features as bitmasks. Do you need a probabilistic outcome? Etc.
Generally you start with the easiest algorithms in your toolbox to see how viable they are. For a classification task I'll almost always start with a naive Bayes classifier (if the data allows) and/or a random forest and see how they perform. If the problem domain is highly non-linear you might start with a support vector or kernel method. Neural network is a last resort for me, as I find most classification problems can be solved to a high accuracy much more simply.
> how and why did you massage the data in a specific way?
This relates back to #1 -- you should only massage data based on what your feature engineering tells you to do. Sometimes you might want to remove outliers or clean up the training data, but only if the outliers really should be removed from consideration entirely.
> There are various feature engineering and feature extraction techniques. Filter methods, wrapper methods, and embedded methods. Principle component analysis, autoencoding, variance analysis, linear discriminant analysis, Gini index, genetic algorithms, etc -- the feature selection process will depend on the dataset, the problem domain, the analysis algorithm you ultimately use, etc.
Obviously thats a big toolbox and Im sure it takes time to develop an intuitive understanding for all these techniques. What I hope for is some sort guidebook on what to look for when I stumble across problems. So lets say you try out an algorithm and your accuracy(or whatever evaluation criteria you might have) is low. How do you figure out if thats due to the algorithm, or is it due to (or due to the lack of) feature selection?
An analogy that might be useful is, when I see my database queries are slow, I can use EXPLAIN to guide what knobs to tune. Obviously it requires understanding what indexes are, what a full table scan is etc. etc. but the EXPLAIN plan provides a guidebook of sorts.
Edit: fix autocorrect
To preview: http://todotemplates.com/posts/HRtYanEq8zMgRL5fz/google-ml-c...
To import directly: https://todoist.com/importFromTemplate?t_url=https%3A%2F%2Fd...
Billions of courses, web sites, job applications and HN posts. The subject seem to have taken off massively in the last two years. I mean image and speech recognition is pretty cool (when it works!), but hardly that earth shattering, is it?
ML is a much broader field than just neural networks. The hype for ML, in general, I think is warranted. We hit an inflection point when AWS launched and scalable processing power became cheap. It became cheap to process tons of data and generate insights. I don't have hard numbers on this, but probably 90-95% of machine learning used in practice is NOT neural networks, and have accuracies in the 90%+ arena. So ML in general -- sure, hype warranted.
Neural networks are the new hot topic, and the hype isn't fully warranted yet. TensorFlow made them very popular in the developer community; this is a good thing because it's spurring more investment and research in ANNs. But for any given problem, odds are that a neural network is not the best (ie, most accurate or cheapest) way to solve it. Neural networks do have specific problem domains where they are the state of the art, but for most other problem domains there exists a better solution. So I'd say that neural networks are a little over-hyped right now, but with a new generation of developers learning about and experimenting with ANNs, that will change in a few years. I think we're about to see an explosion of ANN usefulness over the next few years.
TLDR: ML is very useful but is more than neural networks; neural networks need a little more progress to catch up to the tensorflow hype.
Here's a concrete example: Before spreadsheets existed, there used to be legions of accountants who created complex ledgers on paper and added up all numbers to track how a business was doing. You'd literally mail off your sales numbers to an accounting team somewhere and wait three days to get the latest report generated and sent back. Sure they had calculators to add numbers, but the computers of the day didn't understand how those numbers related to each other. The human still had to do most of the work to create the reports.
The big idea of spreadsheets was to make the computer manage the more complex task of knowing how different numbers in a report related to each other. It made most ledger tasks totally automatic once the initial report was defined. Now a single accountant could do the work of the entire accounting team - and more accurately and in less time! There were stories of the first spreadsheet testers having to delay mailing back their financial reports by a few days because their clients would be suspicious if they mailed them back too fast.
Nearly overnight accounting got a lot more efficient and companies made more money. T"What If" modeling that used to be too slow and cost prohibitive to do was now it was quick and easy. Companies could plan more intelligently. The spreadsheet was a true game changer.
This same pattern happens every time the bar is raised on the complexity of what can be automated and Machine Learning raises the bar one giant notch. Previously we were limited to automating tasks that a smart coder could describe as discrete steps in code. But with ML, the computer can figure out it's own rules just by looking at data. That means in many cases you can solve very hard problems just by collecting a lot of data. Lots and lots of things that used to be done by large groups of people will now be able to be done with a single computer.
In that sense, ML is a total game changer. Don't focus on the specific applications thus far. Focus on the idea that all kinds of tasks that used to require humans can now be automated with a little bit of applied ML. The opportunities are literally everywhere.
In a few years, ML won't be some esoteric technique used by a few people. It will be a core skill that everyone uses or touches in some way. It's going to creep into everything everywhere because it's just so darn useful.
The advances made by deep nets in signal processing will likely slow down now, but they aren't going away, not in the foreseeable future.
The hype around DNNs arose when we took our unbridled enthusiasm for what's they've achieved in DSP and extend it to other domains with data that's less 'dense' and thus aren't as amenable to de/convolution in N-D space or time.
Will DNNs revolutionize or introduce all the techniques needed to achieve AGI/Strong AI? I very much doubt it. As yet, there's little sign that DNNs can perform relational operations on interdependent symbols, like the transforms available via type theory, bayesian nets, or predicate logic.
The multitude of disparate facts and semantics in a rich knowledgebase can't be organized into dense matrices the way that continuous signals can, so the SIMD operations that are so effective in DSP won't implement the rich transformations needed in a relational fact-based knowledge space equally as well, if at all. Thus DNNs almost surely aren't going to take us to the heights of logical or compositional thinking that human level intelligence requires.
But how far up relational mountain will DNNs take us? I suspect that won't be known for a decade or longer. But even if we don't reach the summit, it'll be higher than we were before.
The rate of advance using deep nets in signal processing will likely slow down now, but they aren't going away, not in the foreseeable future.
The hype around DNNs arose when we took our unbridled enthusiasm for what's they've achieved in DSP and extended it to other domains where data is less 'dense' and thus aren't as amenable to fast de/convolution in N-D space or time.
Will DNNs revolutionize or introduce all the techniques needed to achieve AGI/Strong AI? I very much doubt it. As yet, there's little sign that DNNs can perform relational operations on interdependent symbols, like the transforms available via type theory, bayesian nets, or predicate logic.
The multitude of disparate facts and semantics in a rich knowledgebase can't be organized into dense matrices the way that continuous signals can, so the SIMD operations that are so effective in DSP won't implement the rich transformations needed in a relational fact-based knowledge space equally as well, if at all. Thus DNNs almost surely aren't going to take us to the heights of logical or compositional thinking that human level intelligence requires.
But how far up relational mountain will DNNs take us? I suspect that won't be known for a decade or longer. But even if we don't reach the summit, we'll be considerably closer than we were before.
I built something similar (albeit for a relatively limited database of recipes) for a hackathon a couple of weeks back. I didn't even use a proper NLP library, just some simple hand-rolled pattern-matching, and got pretty good results.
Good luck!
The approach was to tokenize the input and then do basic pattern-matching on it, with separate dictionaries of quantity units (e.g. cup, oz, pound) ingredients, processing words (e.g. "chopped") and throw-away words (e.g. "of"). In fact, possibly the most complicated part was parsing "2.5", "2 and a half" and "2½" all to the same thing.
For you actual question, yes, as others have said it might be just an NLP/regexp problem. Otherwise, you could look at ingredients identification as a classification approach. I recommend checking FastText, NLTK, familiarize yourself with word dictionaries and pre-trained vectors that are available, these tools might help generalize your work beyond the data you have at hand.
(E.g. if it works well on your data using pre-trained word vectors from wikipedia, chances are it might work on examples you don't even have.)
You can certainly use high-level ML tools (like Keras), but it takes a great deal of work to wrangle your data into a usable format, and even more knowledge to debug an ineffective network.
Why is that?
"Reducing Loss: Gradient Descent"
is
"Convex problems have only one minimum; that is, only one place where the slope is exactly 0. That minimum is where the loss function converges."
The first sentence is flatly wrong: E.g., for positive integer n and the set of real numbers R, function f: R^n --> R where for all x in R^n f(x) = 0, f is convex, concave, and linear, and for all x in R^n x is a minimum and a maximum of f.
Can there be uncountably infinitely many alternative minima for the Google ML problems? Yes, e.g., just enter one of the independent variables twice.
The second sentence is nonsense.
Grotesque, outrageous incompetence!!!!
It has long been known that minimizing a convex function, even a differentiable convex function, with just gradient descent can be just horribly inefficient. A LOT is known about how to do much better than just gradient descent. E.g., there is Newton iteration (right, that Newton, hundreds of years ago) and quasi-Newton. And there's more.
Why so inefficient? Well, draw a picture like Google did except use just two independent variables instead of just the one in the Google picture. Then see that the resulting, convex "bowl" can be like a long, narrow boat with a very gentle slope in one direction and a very steep slope in an orthogonal direction. Yes the cross section of the bowl can be, first cut, an ellipse with one short axis and one long one. Sure, the axes are eigenvectors, etc. and the ellipse is part of a local quadratic approximation. Well, gradient descent keeps going back and forth nearly parallel to the short axis of the ellipse and making nearly no progress on the long axis. People have known this and known good things to do about it for, uh, at least half a century.
For the Google ML problems, might (1) tweak Newton iteration to improve the rate of convergence to a minimum or (2) at each iteration don't get a gradient descent but get a supporting hyperplane of the epigraph of the convex function, as the iterations proceed, accumulate these hyperplanes, notice that they lead to an approximation of the full epigraph, and use linear programming or some tweak of that to minimize the hyperplane approximation to the convex function -- there is much more that can be said here. By the way, the convex function for that ML problem is quite special, e.g., quadratic.
A lot has long been known and well polished in regression and classification, e.g., with TeX markup:
N.\ R.\ Draper and H.\ Smith, {\it Applied Regression Analysis,\/} John Wiley and Sons, New York, 1968.\ \
Leo Breiman, Jerome H.\ Friedman, Richard A.\ Olshen, Charles J.\ Stone, {\it Classification and Regression Trees,\/} ISBN 0-534-98054-6, Wadsworth \& Brooks/Cole, Pacific Grove, California, 1984.\ \
C.\ Radhakrishna Rao, {\it Linear Statistical Inference and Its Applications:\ \ Second Edition,\/} ISBN 0-471-70823-2, John Wiley and Sons, New York, 1967.\ \
A good start on convexity is
Wendell H.\ Fleming, {\it Functions of Several Variables,\/} Addison-Wesley, Reading, Massachusetts, 1965.\ \
Total, fun, dessert ice cream on convexity is Jensen's inequality; right away can use it to prove a lot of classic inequalities.
Gee, look on the upside!!!! From this sample, the claims of machine learning (ML) revolutionizing the economy are nonsense!!!! And for startups, don't much have to worry about serious competition from Google!!!!
(Full disclaimer - I did theoretical physics, so understand both sides. :))
>function f: R^n --> R where for all x in R^n f(x) = 0
This hyperplane is not convex. A convex curve by definition can not be equal to its tangent at any point.
Edit: I should specify, I mean a convex curve cannot be completely equal to any of its tangents, obviously it will equal each tangent at a single point.
I'm going by the definition of convex function given in Rudin's "Principles of Mathematical Analysis", Apostol's "Calculus", Wikipedia, and MathWorld.
No, my math is fully correct, and your claim is wrong.
For a lecture Convexity 101, see my
It's simple: A cube has supporting planes at each point that is an edge or corner, but those points do not have tangents.
i am certain that you are familiar with the "usual" statistics sequence. (for others: there are lower-division courses that use calculus in a few places but otherwise avoid it, focusing instead on memorizing procedures. there is the upper-division probability/math stat sequence that uses calculus heavily but avoids analysis. and there is an intro phd sequence that finally gets into measure theory.) if you look at a coursera course that gives an high-level overview of practical statistics and simplifies its presentation to be accessible to people who never took calculus, you can criticize the very idea of a course that does not explain the measure-theoretic issues, but it makes no sense to use it to criticize the field of statistics, or to criticize the competence of others in the institution that produced it.
here, google is producing introductory training materials for developers. many developers have never taken calculus, let alone optimization, statistics, or analysis, and when i took MLCC internally, you were supposed to go through this whole thing (lectures and coding) in two days. it's supposed to give you enough understanding of the concepts to understand the API and apply it.
The Google statement I quoted was flatly wrong. It is really important for students to be told that.
I gave some references to more in statistics.
> the measure-theoretic issues
I didn't mention measure theory, and the statistics references I gave don't mention measure theory either. I referenced Fleming only as background on convexity, and there is no measure theory in that part of Fleming.
You mentioned the role of calculus for the math of regression: That use of calculus, for deriving the normal equations, is neither necessary nor, really, sufficient. There is another derivation, nicer, fully detailed mathematically, with no calculus at all. The core of the idea is that the minimization of the squared error has to be an orthogonal projection and, then, presto, bingo, get the normal equations. And there are more advantages to that derivation. To keep my post simple, I omitted that derivation, but a good treatment of regression without calculus could use it.
I omitted the standard definition of convexity, but apparently in this discussion we need that. By omitting the definition, the Google material was not so good.
Definition (convex function): For the set of real numbers R and a positive integer n, a function f: R^n --> R is convex provided for any u, v in R^n and any a in [0,1] we have
f(au + (1-a)v) <= af(u) + (1-a)f(v)
So, for a picture, on the graph of f(x), we have two points (maybe not distinct) (u, f(u)) and (v, f(v)). Then we draw the line between these two points. The number a determines where we are on that line. With a = 0, we are at point (v, f(v)). With a = 1, we are at point (u, f(u)). Then as we move a from 0 to 1, we move along that line. We also have on the graph the point, say, P
(au + (1 - a)v, f( au + (1-a)v ) )
And on the line we drew, we have point, say, Q
(au + (1 - a)v, af(u) + (1-a)f(v))
Well, we are asking that point Q be the same as point P or directly above point P. That is, the line we drew is on or above the graph of (x, f(x)). The line is sometimes called a secant line and is said to over estimate the function.
Definition (concave): The function -f is concave if and only if the function f is convex.
As in Fleming, a convex function is continuous. Intuitively, the proof is based on two cones, and as we approach a point we get herded between the two cones. The cones are from the convexity assumption. Draw a picture.
IIRC, there is a result in Rockafellar that a convex function is differentiable almost surely with respect to Lebesgue measure, but this is the only connection I would make with convexity and measure theory.
For convex f, the set of all (x, y) where y >= f(x) is the epigraph of f, that is, the region on or above the graph of f.
Definition (convex set): A subset C of R^n is convex provided for any u, v in C and any a in [0,1] the point
au + (1-a)v
is also in set A.
Well, as in Fleming, the epigraph is convex. It is also closed in the usual topology of R^n.
Definition (closed set): A subset C of R^n is closed (in the usual topology of R^n) provided for any sequence x_n, n = 1, 2, ... in C that converges to y in R^n, y is also in C.
In particular, if we define the boundary of C, set C contains is boundary. So, the interval [0,1] is closed and the interval (0,1) is not closed.
Well, for any closed convex set and point u on its boundary, there exists a hyperplane that passes through point x and where set C is a subset of the closed half space on one side of the hyperplane.
Such a hyperplane is said to be supporting for set C at point x.
Intuitively, in R^3, push convex set C to be in contact with a wall. Suppose point x on the boundary of set C is in contact with the wall. Then the wall is a supporting plane for set C at x, and set C is a subset of the room side of the wall.
Or think of a big, solid, irregular piece of cheese and John Belushi as his Samurai Tailor swinging his sword: John keeps swinging his sword in arcs that are in flat planes and cuts down the irregular cheese to a convex hunk. So, the convex C has been determined (formed) from the supporting hyperplanes from Belushi's sword.
For another way to make a convex set, take a piece of wood and press it against a belt sander until the boundary consists of only flat sides.
Let a solid rock roll around in a steam of water for a few thousand years, and may end up with a smooth, shiny, convex rock.
Faster a chicken egg is convex.
In general, a closed convex set is the intersection of its supporting hyperplanes.
Then, we can approximate a convex set with some of its supporting hyperplanes. In particular, we can approximate a convex function with some supporting hyperplanes of its epigraph. At times, this can be useful -- it's the main idea behind Lagrangian relaxation in constrained optimization (I used that once).
In particular, the epigraph of a convex function is the intersection of the supporting hyperplanes. In that case, a supporting hyperplane is called a subgradient. The function is differentiable at the point of contact if and only if the subgradient is unique. If the subgradient is unique, then it is just the tangent hyperplane from the gradient of the function.
In R^3, a cube is a convex set. Each of its sides is part of a supporting hyperplane. For a point on the boundary of the cube that is not on an edge, the supporting hyperplane at that point is unique. The corners and edges of the cube also have supporting hyperplanes, but they are not unique.
In R^3 a sphere is convex. Then it is also the intersection of its supporting hyperplanes. At each point on the boundary of the sphere, the supporting hyperplane is unique.
Similarly for epigraphs.
So, the function f: R*n --> R where for each x f(x) = 0 is convex, concave, and linear, and each x is both a maximum and a minimum of f. So, the minimum of a convex function need not be unique. In this case, the epigraph is just a closed half space.
"Look, Ma! No calculus!" And no measure theory.
Exercise: Derive the regression normal equations via perpendicular projections and without calculus.
Exercise: Argue the role of perpendicular projections in the minimization in regression.
i made my request because it's jarring for me as a reader when you punctuate your (often delightful) expository writing with conclusions about entire fields and large organizations that seem (on the face of it) to be justified by old and/or very limited data.
but that's a selfish request, and you are of course free to tell me to get lost and post whatever you want (and i'll still read it); i'm certainly not going to pursue this further, aside from the apology and clarifying comment above.
If they are going to plagiarize, even just teach, regression analysis, then at least they shouldn't make a mess out of it, and a mess is what they made. Google should "get that MESS OFF the Internet".
Students should be told the truth: Regression is powerful stuff. Sometimes the results can be valuable. The Google OP is an introduction to regression and does have some value. But the Google material is a MESS, and students should be informed that they are getting really low quality material and should see some references to some beautifully polished material.
So, I helped any students who would be the target audience for the Google OP.
You should know this; I believe you do.
I'm offended by the mess and passing that out to students trying to learn. You should also be offended.
> So, the function f: R*n --> R where for each x f(x)
should read
So, the function f: R^n --> R where for each x f(x)
Excuse: Just now I'm using a keyboard on a laptop, and I'm not used to the keyboard yet.
For implementing exercises using Java, you have a bunch of good options:
1) The most direct equivalent to Pandas+Tensorflow I can think of is DL4J. They have a good comprehensive set of concept and implementation tutorials [1].
2) TF APIs have a Java port and can be used from java desktop and console applications [2]. So a second but slightly more difficult option is using TF Java port + Spark APIs.