Hacker News Reader

Comment by hypersoar | Hacker News Reader

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

View on HN

For non-deep learning, read David Barber's book:

http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf

Some sections may be less relevant, depending on what you want to do, but Section III is a very good introduction to machine learning methods.

Do the exercises as you're reading. Theory is one thing, but in ML my rule of thumb is that you don't really understand a model until you've coded it up. A collection of written exercises would be a good way to impress an interviewer, too.

Note: 090310.pdf (9 March 2010) is an older version of the textbook. The latest revision of the textbook PDF (091117.pdf, 9 November 2017) is available at:

→ http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/091117.pdf

Aside:

⸰ Textbook homepage: http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=...

⸰ Online version homepage (should always contain link to latest revision): http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=...

⸰ Directory sorted by date: http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/?C=M;O=D

It really is a great book. However my friend & I attempted to make it through and faced quite a bit of struggle. Eventually we'd figure things out but it felt like it would of been significantly easier with the help of a teacher/mentor to ask questions to. Unfortunately it's actually kind of hard to find colleges nearby that even have courses in Bayesian statistics (from an inference perspective). That was frustrating.

It's far from state of the art, but reading Computer Vision by Simon J D Prince [1] alongside David Barber's BRML can really help.

[1] - http://www.computervisionmodels.com/

I had this book on my computer back from when it was released, but never got around to reading it. Thank you for the recommendation!

For Machine Learning, I found "Learning From Data" http://amlbook.com/ to be a very strong, foundational book. It does the best job of any reference I've seen of answering the question: "Is learning possible and feasible?" Provides and excellent mathematical foundation for understanding when and why ML systems should succeed and fail. Associated MOOC from Caltech at https://work.caltech.edu/telecourse.html

My ML professor used this book, and I loved it. Some presentations of ML make it seem like a bag of tricks; this feels like the opposite of that.

I'm a big fan of the MOOC, even though it didn't explicitly cover it, it basically explained what people mean when they say "we don't know why neural nets work", which is that our previous generalization bounds would imply that neural nets would not generalize well, but in practice they do.

If you have at least some coding experience and you are interested in the practical aspects of ML/DL (i.e., you want to learn the how-to, not the why or the whence), my recommendation is to start with the fast.ai courses by Jeremy Howard (co-author of this "Matrix Calculus" cheat sheet) and Rachel Thomas[a]:

* fast.ai ML course: http://forums.fast.ai/t/another-treat-early-access-to-intro-...

* fast.ai DL course: part 1: http://course.fast.ai/ part 2: http://course.fast.ai/part2.html

The fast.ai courses spend very little time on theory, and you can follow the videos at your own pace.

Books:

* The best books on ML (excluding DL), in my view, are "An Introduction to Statistical Learning" by James, Witten, Hastie and Tibshirani, and "The Elements of Statistical Learning" by Hastie, Tibshirani and Friedman. The Elements arguably belongs on every ML practitioner's bookshelf -- it's a fantastic reference manual.[b]

* The only book on DL that I'm aware of is "Deep Learning," by Goodfellow, Bengio and Courville. It's a good book, but I suggest holding off on reading it until you've had a chance to experiment with a range of deep learning models. Otherwise, you will get very little useful out of it.[c]

Good luck!

[a] Scroll down on this page for their bios: http://course.fast.ai/about.html

[b] Introduction to Statistical Learning: http://www-bcf.usc.edu/~gareth/ISL/ The Elements of Statistical Learning: https://web.stanford.edu/~hastie/ElemStatLearn/

[c] http://www.deeplearningbook.org/

There are many new methods cropping up that most people in the data science hype train will be full-on unable to access, including methods on manifolds (even kernel methods on manifolds) and algebraic-topological methods (persistent homology) with enough maths to give Kagglers the screaming meemies.

I'm using some of those for $(redacted, the plan is to make money). Don't follow the crowd.

When I started out in ML I was really keen to learn about the most 'mathsy' approaches out there.

I think with hindsight, it's great to have a broad spectrum of methods available to you, but if you focus too much on methods at the hard-math end of the spectrum just for the sake of an intellectual challenge, you can end up fixated on an exotic solution looking for a problem while the rest of the field moves on, rather than doing useful engineering people care about.

Maybe you find a niche where something exotic really helps, maybe you don't -- maybe for research this is a risk worth taking. But just something to keep in mind.

IMO: breadth is good. Mathematical maturity helps. If one sticks around one finds uses for interesting maths eventually, but not worth trying to force it.

Another avenue for people who want to use some hardcore math: try and use it to find some good theory around why things which work well, work well. Not an easy task either by any means.

Any papers or books to get started with this? I'm in ML but not on the DL hype train - disappointed by its limits.

Math? Meh.

Just throw more GPUs at it bro. Gradient descent 4ever! xD

I made the same transition earlier in my career. One book on deep learning that meets your requirements is [0]. It’s readable, covers a broad set of modern topics, and has pragmatic tips for real use cases.

For general machine learning, there are many, many books. A good intro is [1] and a more comprehensive, reference sort of book is [2]. Frankly, by this point, even reading the documentation and user guide of scikit-learn has a fairly good mathematical presentation of many algorithms. Another good reference book is [3].

Finally, I would also recommend supplementing some of that stuff with Bayesian analysis, which can address many of the same problems, or be intermixed with machine learning algorithms, but which is important for a lot of other reasons too (MCMC sampling, hierarchical regression, small data problems). For that I would recommend [4] and [5].

Stay away from bootcamps or books or lectures that seem overly branded with “data science.” This usually means more focus on data pipeline tooling, data cleaning, shallow details about a specific software package, and side tasks like wrapping something in a webservice.

That stuff is extremely easy to learn on the job and usually needs to be tailored differently for every different project or employer, so it’s a relative waste of time unless it is the only way you can get a job.

[0]: < https://www.amazon.com/Deep-Learning-Adaptive-Computation-Ma... >

[1]: < https://www.amazon.com/Pattern-Classification-Pt-1-Richard-D... >

[2]: < https://www.amazon.com/Pattern-Recognition-Learning-Informat... >

[3]: < http://www.web.stanford.edu/~hastie/ElemStatLearn/ >

[4]: < http://www.stat.columbia.edu/~gelman/book/ >

[5]: < http://www.stat.columbia.edu/~gelman/arm/ >

Goodfellow book [0] is available for free, http://www.deeplearningbook.org/

+1 for Gelman, but I hate Bishop's book [2]. It was an early go-to reference in the field, but there are better books out there now.

What do you hate about Bishop's book? I'm genuinely curious.

Honestly, I don't understand the way he explains things. The maths is difficult to follow, and it just never clicks for me. Maybe he's writing for someone with a physics background or something, but I feel stupid when I read bishop.

I just read over his description of how to transform a uniform random variable into a variable with a desired distribution (p. 526). It's a fairly easy trick, but if I didn't already know it I wouldn't understand his explanation

I'm trying to read through it and I have to agree, his math isn't that clear to me. What do you recommend?

David Barber!

Sure, Goodfellow book

http://www.deeplearningbook.org/

Given your math bent expecially analysis, I would recommend you start with Luc Devroye, follow it up with Vapnik. These are a bit dated but still very relevant. After these two go with Alex Smola and Bernard Scholkopf. The books that have been suggested to you so far aren't bad but with high fluff to meat ratio, not meant for a reader like you. If you have function analysis in your bag you will feel right at home with the books I suggested. Deep NN are much in the news, but on the math side of the practice of DNN, there is not much to it apart from chain rule for differentiation.

The book "A Probabilistic Theory of Pattern Recognition" by Devroye, Györfi and Lugosi is available from the site of Gyorfi at http://www.szit.bme.hu/~gyorfi/pbook.pdf .

Thanks for the recommendations. Devroye wrote several books, and I assume the one being mentioned is "A Probabilistic Theory of Pattern Recognition".

Yes that's the one I had in mind. His other books are also very good, but are more specialized.

You may also enjoy graycat's comments on HN. He know his math but is contrarian about machine learning but its good to have that view point.

Are you only interested in ML practicum? What did you take after real analysis?

What are your thoughts/interests on analysis for ML, like the approximation theory that branched from Fourier and Wavelet analysis catching on with Cybenko [1], continuing with among others Mhaskar [2], most recently added to by Bolcskei [3]. And then there are other areas where analysis applies like studying the optimization of such networks...

(And of course, this is just for NNs. There are other areas of research where analysis comes into ML. And of course, real analysis lays the foundations for probability and statistics, and is not abstracted away in many research areas in these fields.)

[1](https://pdfs.semanticscholar.org/05ce/b32839c26c8d2cb38d5529...) [2](https://pdfs.semanticscholar.org/694a/d455c119c0d07036792b80...) [3](https://arxiv.org/abs/1705.01714)

https://github.com/llSourcell/Learn_Machine_Learning_in_3_Mo...

This is also a good resource for MachineLearning

You don't need much CS knowledge for deep learning, in fact pure math/stats is more useful. I'd say CS knowledge is only useful if you do RL stuff and even then you don't really need it that much.

That really depends on what you're doing. I've seen people coming from a pure math background really struggle. If you've never programmed before, then using something like TensowFlow isn't exactly going to feel natural. Familiarity with Python and the Linux ecosystem would definitely be quite helpful. Yes, I'm sure you could run ML models on a Windows machine, but at some point, you might want to perform experiments on a cluster, and it's going to be running Linux.

True. I was assuming programing experience but no data structure and algorithms. I can attest that a lot of people in my ML class couldn't program and they sucked at the assignments we had.

Tell that to my potential employers :|

Depends what jobs you're applying to. But I've seen it first hand traditional HR doesn't know how to filter for ML resumes.

Reply on news.ycombinator.com