Learning Theory from First Principles [pdf]
di.ens.fr
di.ens.fr
And then "deep learning" arrived and it became even more obvious that the only thing that matters is data and time spent crunching numbers (more of both and you get better results no matter the model).
Again I just want to be crystal clear, because I'm sure someone will pop in and claim "oh I still use SVM to pick my family's shopping list": no professional ML engineer/team/org today that ships and ML product "at scale" gives a fuck about SVMs or graphical models or bayes nets or kernel methods. No one. So who cares about all this sophistry? What value is it to learn concentration inequalities - training goes brrr no matter what if you have enough data. And if you don't, if you're really building a model to predict your family's shopping list, I encourage to reflect on whether it would be simpler to just ask your family what they want for dinner instead.
My 2 cents: teach people/students useful things instead of this stuff. They'll be happier and you'll feel more fulfilled (even though you didn't get flex your big math brain).
Francis Bach, the author, makes a good faith effort to explain exactly why this material is beneficial (see https://francisbach.com/my-book-is-out/):
"Why yet another book on learning theory? ...the main reason is that I felt that the current trend in the mathematical analysis of machine learning was leading to overly complicated arguments and results that are often not relevant to practitioners. Therefore, my aim was to propose the simplest formulations that can be derived from first principles, trying to remain rigorous without overwhelming readers with more powerful results that require too much mathematical sophistication."
From my own reading and experience on the mathematical analysis approach of this "training goes brrr" approach, I thought the material in Chapter 12, Overparameterized Models, was interesting and coherent with 12.2.4 Linear Regression with Gaussian Projections being an especially elegant explanation. It would be interesting to hear if you had read/skimmed/purused this section and found it wanting etc.
I understand that deep learning is all in vogue now. But when I was in graduate school, a professor asked me why I was using neural nets in a project since it was not as good as SVMs. We used to study Vapnik and VC dimensions, SVMs etc. and neural nets were totally out of fashion.
Imagine what would have happened if everybody were using and researching only those methods because they worked better. And deep learning could benefit from a theory that explains why, when and how it works so well. Maybe someone working on this could develop on it to include it.
Also I don’t think you’re right to assume that all models out there are deep learning models. Yes they are very good for many cases (specially those with less structured data, like image or nlp). But in some cases gradient boosting or even GLMs are better suited for the task (because of the structure and size of the data or because of computing restrictions).
And in the end, people can just want to learn it because they find it interesting. It’s a bit sad to do only things that are “useful”. That’s my 2 cents.
Teaching others how to replicate solutions is very different from guiding people how to solve yet unsolved problems in the field.
This book is for the latter. For the former, you might want to look out for one in the "for dummies" series.
My guy I learned this material from a healthy mix of ESL and Casella Berger and Billingsley. I could still, to this day, probably do every proof in this book without reviewing the material. And yet, despite all that training, I still argue this book is not useful for absolutely anything except assigning homework problems and setting exams.
resources like this are useful, in an academic setting. lets not forget we all forget 50% or more of what we learn about 20 minutes afterwards unless we consistently remind ourselves of what we learn.
unless others prove me wrong using personal anecdotes.
There’s a reason why “AI is just statistics” became a meme: a lot of places do use textbook machine learning techniques and dress it up as AI. Yes deep learning will win with enough data but few companies have that luxury.
I would like to learn all, specially thing that lead to the current status.
Also, the "big math brain"'s you're talking about probably read all the books your shutting down. I'd say their big math brains are the reason we have LLMs today.
This textbook is hardly irrelevant for people who only care about deep learning though. It covers regularisation, optimisation, overparameterised models, double descent and err, neural networks. Sounds pretty relevant to me?
If you think the rest of the book is irrelevant then skip it.
You sound a bit nutty when you confidently state nobody uses any of the other methods in this book. How could you possibly know that?
Discovering knowledge and using the knowledge that works best are very different.
Your argument remind me from this lecture from Feynman. Quoting him: "...and every theoretical physicist that is any good knows 6 or 7 different theoretical representations for exactly the same physics and knows that they are equivalent... but he keeps them in his head hoping that they'll give him different ideas for guessing."