Hacker News Reader

Comment by supremus_58 | Hacker News Reader

does anyone have recommendation on a text that is mathematically heavy but also looks at modern approaches/appleications?

View on HN

You can order the standard machine learning texts from most to least math-y, and least to most modern:

- Pattern Recognition and Machine Learning (Bishop 2007)

- Machine Learning: A Probabilistic Perspective (Murphy 2012)

- Deep Learning (Goodfellow, Bengio, Courville 2016)

If you want cutting-edge material, read the Deep Learning book (which is still quite technical, though some of its content may be outdated in a few years). If you want timeless mathematical foundations very clearly presented, read Bishop. Murphy is a good middle ground.

If you're self-teaching and have trouble focusing on a textbook for long periods, the Stanford CS229 lectures combined with Andrew Ng's course notes and assignments are probably the best resource. They are still quite rigorous and working through them will give you a solid foundation, after which you'll be more prepared to understand the deeper content in any of the texts above.

Where would you place "Elements of Statistical Learning" in relation to these, if you know?

You want ISLr (http://www-bcf.usc.edu/~gareth/ISL/), not ESL. ESL is the prototype for the former.

Not sure why this is downvoted. ISLR is widely considered the "easier to read" / more pedagogical version of ESL. It's still mathematical but is closer to what the poster wants - a textbook that teaches them.

Interesting, thanks - I wasn't aware of this text.

But according to your link, ESL isn't a prototype of ISL, but a more 'advanced treatment'. The R applications in ISL seem like they might be very useful, though.

ISLR was written after ESL to be an easier to read version. I guarantee you will have trouble if you try to teach yourself ML with ESL.

I haven't read it in detail, but my impression is that it is mathy, like Bishop, but focuses more on 'classical' frequentist analysis, whereas Bishop takes a more open-ended Bayesian perspective and covers important machinery like graphical models and inference algorithms that I don't think are in ESL.

As a researcher I tend to prefer the Bayesian perspective in Bishop, because it gives you a unifying framework for thinking about building your own models and learning algorithms. But lots of people seem to respect ESL and speak very highly about it. It's probably valuable if you are implementing one of the methods it covers and want to understand that specific method in great depth.

I appreciate the comparison. My field is computational biology, and the latest edition of ESL has some specific, relevant examples, which is what drew me to it in the first place. I'm currently working on getting my statistics background up to par, and then I'll be choosing an ML textbook. However, your note about ESL focusing on frequentist models gives me pause about my original choice, as it would seem to me biological applications naturally lend themselves to Bayesian methods. I think in the end I'll have to crack open several books to decide. Thanks for the input.

You're welcome! FWIW I mostly agree with argonaut's point elsewhere in this thread - very few people successfully self-teach ML from a textbook alone. So whichever book(s) you choose, it might also be worth working through some course materials. I've already suggested Stanford's CS229 for solid foundations, but depending on your interests in bioinformatics, Daphne Koller's Coursera course on probabilistic graphical models (https://www.coursera.org/learn/probabilistic-graphical-model...) might be especially relevant. Koller literally wrote the book on PGMs, has done a lot of work in comp bio, and her MOOC is apparently the real deal: very intense but well-reviewed by the people that make it through.

Besides the DL book, there's other excellent texts that are freely available/open content on the web:

- Elements statistical Learning, Hastie et al

- Shalev-Shwartz and Ben-David: http://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning...

- the late David MacKay's Info Theory

- Bayesian Reasoning in ML, Barber

- Hopcroft/Kannan (this is an older version, you can google latest: http://www.cs.cornell.edu/jeh/book112013.pdf

I didn't expect to see Hopcroft/Kannan in that list. I used it in a course taught by Kannan and back then it was called "Computer Science Theory for the Information Age". Apparently the book is now called "Foundations of Data Science". I always thought the old naming was terrible but compared to the new title it actually describes the content a lot better since the book is mainly about CS theory and mathematical foundations for them. It was one of my favorite courses but I would not classify it as ML book.

[1]https://www.cs.cmu.edu/~venkatg/teaching/CStheory-infoage/

[2]https://www.cs.cornell.edu/jeh/book2016June9.pdf

I hear people recommend textbooks a lot, and I honestly don't know why. Very few researchers I know learned machine learning through reading a textbook on their own. Furthermore, the first two textbooks are closer to reference books rather than actual pedagogical tutorials (I haven't read any single chapter in its entirety). The Deep Learning book assumes machine learning knowledge.

What would you recommend instead for a beginner trying to get into ML?

I'll tell you how I started my journey:

I took the Stanford ML Class in 2011 taught by Andrew Ng; ultimately, Coursera was born from it, and you can still find that class in their offerings:

https://www.coursera.org/learn/machine-learning

On a similar note, Udacity sprung up from the AI Class that ran at the same time (taught by Peter Norvig and Sebastian Thrun); Udacity has since added the class to their lineup (though at the time, they had trouble doing this - and so spawned the CS373 course):

https://www.udacity.com/course/intro-to-artificial-intellige...

https://www.udacity.com/course/artificial-intelligence-for-r...

I took the CS373 course later in 2012 (I had started the AI Class, but had to drop out due to personal issues at the time).

Today I am currently taking Udacity's "Self-Driving Car Engineer" nanodegree program.

But it all started with the ML Class. Prior to that, I had played around with things on my own, but nothing really made a whole lot of sense for me, because I lacked some of the basic insights, which the ML Class course gave to me.

Primarily - and these are key (and if you don't have an idea about them, then you should study them first):

1. Machine learning uses a lot of tools based on and around probabilities and statistics.

2. Machine learning uses a good amount of linear algebra

3. Neural networks use a lot of matrix math (which is why they can be fast and scale - especially with GPUs and other multi-core systems)

4. If you want to go beyond the "black box" aspect of machine learning - brush up on your calculus (mainly derivatives).

That last one is what I am currently struggling with and working through; while the course I am taking currently isn't stressing this part, I want to know more about what is going on "under the hood" so to speak. Right now, we are neck deep into learning TensorFlow (with Python); TensorFlow actually makes things pretty simple to create neural networks, but having the understanding of how forward and back-prop works (because in the ML Class we had to implement this using Octave - we didn't use a library) has been extremely helpful.

Did I find the ML Class difficult? Yeah - I did. I hadn't touched linear algebra in 20+ years when I took the course, and I certainly hadn't any skills in probabilities (so, Kahn Academy and the like to the rescue). Even now, while things are a bit easier, I am still finding certain tasks and such challenging in this nanodegree course. But then, if you aren't challenged, you aren't learning.

Thank you for the comprehensive response! I have a lot to learn but it's exciting.

Also consider Foundations of Machine Learning (2012) by Meryar Mohri. It is a very good book with a statistical learning approach to ML. If you're interested by statistical guarantees to ML algorithms, eg. why the SVM algorithm works and its sample complexity (the expected out of sample error in function of the sample size), it's the book you need.

The main reason we see all these relatively information-less blog posts / tutorials is because most people on HN are afraid of the math. If you're not afraid of the math (or have a relatively strong math background), then honestly you can jump into the CS 229 Stanford lecture notes and homework assignments. If that's a little too heavy for you, you can also start with the MOOC version of that course (which is a very watered down version of CS 229).

Reply on news.ycombinator.com