When I was working in that stuff, the usual point about the SSBNs (the US long range missile firing submarines) was that they were darned difficult to find. Also, it was standard to assume that they moved in some variation on Brownian motion so that, even if did get some data on their location, it was not very useful for extrapolation.
So, a major point about the design and operation of the SSBNs was that they be darned difficult to find -- via passive sonar, magnetic anomaly detection, active sonar, or anything else. Also, a big deployment of search resources that might be more able to find an SSBN would be regarded as provocative.
Then there was the point: If the Soviets did find a US SSBN, now what? If they track it for very long, then that is provocative. If they don't track it, then they will soon lose it again. If they sink it, then there are all the other US SSBNs ready to fire back.
So, the Soviets would have to find AND sink ALL the US SSBNs at the same time. Finding all of the SSBNs at the same time without being provocative was regarded as essentially impossible.
So, maybe somehow there could be a nuke war but limited to sea. Then could pick off each SSBN as it was found one at a time. Then how long would the SSBNs last? Once I was asked to answer that! So, I derived some math, wrote some software, submitted my results, and the next day my wife and I went for a vacation in Shenandoah! Later my work was sold to a leading US intel group. I could tell you which one, but then I'd have to ...!
Ah, the math wasn't either artificial intelligence or machine learning!
Nothing like a big bang to scare the population into projectfear.
Both sides spent the entire Cold War shadowing each other constantly. There was nothing provocative about it.
It is applied math, anybody who claims otherwise could not be more wrong. It is applied probability theory and computer science with lot more focus on algorithms than is expected of a statistician, and yes without a fetish for asymptotic normality. Usually, but not always, with a stronger focus on prediction accuracy rather than on accuracy of parameter estimates. In that sense its closer to (non-parametric) decision theory than to run of the mill stats.
Yup, if look carefully, in many common cases can do well on prediction accuracy even when parameter accuracy is not so good. E.g., can even do well on prediction accuracy when parameter accuracy is zilch, e.g., some, but not all, cases of over fitting.
You are right about the applied math: Machine learning has a lot that is good and new, however the good is mostly not new and the new, not very good. E.g., I watched a Prof Ng lecture on the gradient descent unconstrained minimization involved in their fitting, and it seemed that either their minimization problem is awfully easy or Prof Ng didn't know much about the applied math of such minimization going way back.
So, machine learning stole a lot from some now quite classic applied math, assigned some new names, e.g., learning and another anthropomorphic name they dreamed up for iteration step size, and claimed something new. That's called academic theft, plagiarism, or some such, right?
Then there's the hype. So, just from machine learning, the pop culture press is extrapolating to the Terminator movies. Absurd.
Then, bigger, there's the implicit assumption, if only from just playing with the words, that the main path to smarter machines is machine learning. Well, if the effort is limited to Breiman's piecewise linear curve fitting, that path is absurdly limited. Much better for making machines smarter, i.e., do more, work better, is just to start with lots of now classic pure/applied math, get out a clean sheet of paper, and derive what need for the problem at hand. As much as I like Breiman, both his early work and his last work, there're oceans more good work in applied math that is nothing like Breiman's last work, e.g., with high irony, his earlier work!
Uh, here's another case where it looks like a machine has learned -- stochastic optimal control. In some cases, it can look darned smart. Another place -- game theory, to be more specific, the von Neumann-Morgenstern work. Then the Nash generalization. But such things are far from the current machine learning.
I would say screw the hype just read what it is. I think your disdain for the name 'machine learning' is getting in your way.
I strongly disagree with your allegation of academic dishonesty though. Optimization has been a fundamental part of ml from the very start and with a strong overlap in the community. I am sure nemirovskii will ring a bell. In any case how can ml steal from applied math when it is applied math. The lingo differs here and there: estimating parameters become learning parameters etc.
Andrew NGO's course is not for you. Go directly to the books and then conference and journal papers.
Focus varies but info theory, stats, signal processing, approximation theory are pretty much the same thing, so are their open problems.
Btw you could not be more wrong about stochastic optimal control or repeated games being far from current ml. Oh and another major major tool in ml is geometry of function spaces. You are reading the wrong sources, probably the populist ones. Get to the real stuff given what i know of your taste in math i think you will enjoy it.
https://www.youtube.com/watch?v=mbyG85GZ0PI (incidentally for graycat, Yaser Abumostafa is a Muslim Egyptian immigrant from Cairo) He covers the VC dimension in https://www.youtube.com/watch?v=Dc0sr0kdBVI, and leaves the proof to an appendix of the book.
Lucky he got out of Egypt before they strung him up! Such good people is what US immigration has for some decades now tried to be about. Maybe we will get back to it.
"Muslim"? I don't care if he is Zoroastrian either. Or worships some sun god. I don't care about his religion. I do care if he wants to blow up buildings. Somehow I doubt if he does.
Looking at his videos, my first cut, crude guess is that he is looking at modern generalizations of old discriminate analysis. Yup, that can be important. Maybe it could be important for, say, one of my old interests, anomaly detection, say, as a doable alternative to Neyman-Pearson where often in practice we don't have nearly enough data. Maybe his interest is in medical diagnosis which, IIRC, was some of Breiman's interest.
But, first cut, it looks like, again, the criterion will be, does the model fit the data well? That is, we have little or nothing to recommend the model except that it fits the data well. But, then, in the case of his lectures, it looks like maybe he is making progress to also knowing that the model will predict well. I'm looking forward to how he does that.
In contrast, if that is important, in my work in anomaly detection, discussed here on HN often enough, I found false alarm rate from some derivations in applied probability with no model fitting at all. Okay, I don't care if the cat is black or white as long as it catches mice.
From a glance, it looks like he is addressing what is meant by learning -- terrific! Not just throwing words around! Then he seems to be addressing when such learning is feasible, etc. Sounds good; I've wondered some about something like that.
But my interest now in what he is doing is a bit limited since the core math in my startup seems to be quite different.
Thanks.
Yes that's exactly it. By way of Vapnik and Chervonenkis' result (essentially an uniform law of large numbers) one upper bounds the expected accuracy (over the unknown distribution) of a classifier in terms of the training error and another quantity that depends on the class of hypothesis that one is using. One can give bounds even when one is using an infinite class, for example all linear functions in the feature space, or some Hilbert space of functions.
This was one of _the_ major early break through result. Its often quoted in the context of ML but it really is a result in probability theory. Since it bounds the most pessimistic situation possible, they are quite bad (although achievable).
It also brought about a paradigm change in the mindset. Since the optimal classifier is just the thresholded conditional density, early approaches had focused mostly on estimating this conditional density. But that's an impossible task. V&C showed even if you do not have enough data to learn the density, you may have more than enough for good prediction accuracy. Don't learn the conditional density, just learn the discriminating function directly by optimizing its expected loss.
People have moved to different tools to bound expected prediction accuracy. You get a lot more reasonable bounds, say with the PAC-Bayesian theorem.
Key thing is that these are distribution independent, non-asymptotic and also dimensionality independent.
"glivenko cantelli on steroids", good. Sounds like they actually did something.
Yes, I'm torqued by the new learning labels on old bottles of pure/applied math, but that is not in my way.
> The lingo differs here and there: estimating parameters become learning parameters etc.
Rubs my fur the wrong way.
If they have some stuff beyond borrowing from Breiman, okay.
What's "in my way" now is my startup: I've got the math derived and typed into TeX and the 80,000 lines of typing for the code, with the code running, intended for production, and in alpha test, so just for now I no longer have any pressing math problems to solve.
But, in time, I may return to my math and tweak it a little to try to get some variance reduction. Maybe some of the better machine learning literature would help, or maybe I'll just derive it myself again.
Function space geometry is about where much of my core math is.
Thanks.
Happy to hear back from you. I am actually gladdened that your anomaly detection work is getting some interest on HN lately. Hope something comes out of it. I am now slowly coming to the conclusion that pushing better methods on an existing stack would be really hard. Too much friction, too much politics. Perhaps the way is to create your own better cloud of servers, but that's really big league stuff. Not sure I have the stomach for that.
Curious if you have given any thought to the choice of the metric space where you define your statistics. That might play an important role from what I have seen. There might be an interesting manifold story there.
Big spoilsports are non-stationarities and even bigger are those fat tails. If only everything had a moment generating function.
I see that you have been pointed to Abu-Mostafa he is definitely a good source. Not that Andrew Ng is unaware of the stuff, far from it, he is fighting a different battle: to make parts of ML a commodity.
If you have time then you can browse the proceedings of COLT (conference on learning theory) and ICML.
> or maybe I'll just derive it myself again.
You almost always have to derive it yourself anyway even after you have seen the derivation by somebody else.
As to on topic discussion - wouldn't you only need to take out the comms? Then the fellow subs wouldn't know anything was amiss.
Taking down several redundant, military grade links doesn't sound easy, but finding every sub in the ocean seems much harder.
Maybe there is a watchdog / normal coded communication/expected updates, so it would have to be more of a MITM thing. Even in the worst case, it would leave a limited window of time to act. Not to mention the submarine would likely be forced to try and hail "ground control" which would make DF + geo much easier...
OTOH, a nuke doesn't need to yield several megatons. First devices were about 20 kilotons and it's possible to create smaller ones. I'm sure I've seen some video of 1 kiloton bomb. Also, intuitively, it doesn't seem that it would destroy a sub a few miles away looking at Baker test:
https://www.youtube.com/watch?v=5nWFx-zmI0k
Around 1:25 there's a brief fragment showing the explosion underwater with the usual expansion-contraction cycle.
The maximum depth for a https://en.wikipedia.org/wiki/Mark_48_torpedo is estimated at 800 meters.
Where did you get this idea from? Submarines can launch torpedoes at whatever depth the torpedo itself can operate at.
Movies. On second thought, there was some submarine against submarine battle in Red October... most other submarine wartime movies were from WWII, always shooting at boats and at periscope level to aim.
Oh, and speaking about wars, relevant nick :-)