Basics of Machine Learning

No, there are assumptions also for this approach, and these assumptions also need to be checked.

Then in applying the model that has been checked, there are still more assumptions, that need to be checked.

And checking models is often essentially hypothesis testing where need more assumptions and, commonly, some experience to know what hypothesis tests to use. E.g., tests in regression can be F ratios and/or t-tests, and need to check some assumptions or at least robustness here.

E.g., in power spectral estimation, see, Blackman and Tukey, 'The Measurement of Power Spectra', there is a severe trade off between resolution and stability, another case of a trade off between bias and variance. Tough to 'automate' that consideration.

So, take a box of data. Divide it in half. Use the first half for the 'fitting' or 'training' data. Test on the second half. Not deep; I advised some people in finance to do that in about 1982; not nearly new; and I very much doubt I was the first; gotta believe that J. Tukey, L. Breiman, and others thought of this long before I did.

And if the test fails, then try to fit again. Run all night and in the morning, presto, have a model that does well on the test data. Now what have we? Not a trivial question to answer.

Why least squares instead of something else? A good answer is not easy.

I'm not criticizing biology, and it's done some excellent science, e.g., see the E. Lander lectures in his MIT course in biology.

Statistics is a good field with a lot of good work by a lot of bright people with excellent backgrounds in pure and applied math.

The first time I heard about 'machine learning' I guessed they meant filling in the optimal value function for stochastic dynamic programming. Okay, that is a kind of 'learning'. But, that's not what they had in mind.

Next an example was the children's interactive computer game Animals where a child thinks of an animal, answers some questions about the animal to chase down a binary tree, stored in the program, to a leaf, and have the computer guess the animal. If the computer is wrong it asks the child "What is true for your animal and false for my guess.", adds to the binary tree, and thus, 'learns'. Call it a parlor trick.

A little like 'self driving cars': Really 'self-driving'? Nope, not close. Only on streets where everything has been mapped down to 1 cm or better including all the painted lines, all the curbs, all the traffic signs, and all the traffic lights, etc. In no way does the 'self driving car' actually look at a new street scene, make sense out of it, and use that 'learning' to drive. Instead, so far a self driving car is about as amazing as a train on a track.

We're talking hype, old wine, filtered, in new bottles, and not much that is new and good. Or, the old academic joke goes, "the new is not good and the good, not new.". The last time I looked at a leading 'machine learning' prof, I had conclude that he needed to return to ugrad school, be a math major, and learn how to read/write math.

E.g., why maximum likelihood estimation? There are reasons, but I saw no hint of them in the Ng lectures at Stanford.

We've known a lot about pure and applied statistics and how to use them for a very long time. The good work in statistics is high quality pure/applied math, e.g., Billingsly, 'Convergence of Probability Measures', e.g. with Ulam's result Le Cam called 'tightness', Brillinger on time series, Serfling on limit theorems, Rao on linear methods, Breiman on CART, and much more. I'm not seeing comparable quality in 'machine learning'; indeed, from all I've seen only a tiny fraction of the computer science machine learning people have the math prerequisites for good research in statistics.

Yes, I've published peer-reviewed original research in statistics, indeed, applied to a problem in computer science.