Hacker News Reader

Comment by ilzmastr | Hacker News Reader

Happy to see a book like this trending on hn, especially with a sentence like: "Using fancy tools like neural nets, boosting, and support vector machines without understanding basic statistics is like doing brain surgery before knowing how to use a band-aid." in it's preface. I definitely agree, since I wasted a lot of time doing fruitless surgeries before I went and learned about band aids in depth.

From my look at Part 1, it has some great coverage of the basics, all of which are important. Some of the fundamentals that I see left out are rightly left out since they require experience in real analysis to appreciate, and maybe aren't very actionable. There's few proofs, but, since the goal is a quick understanding, I can also appreciate this.

It looks to me like a great intro of statistics for CS people, as the author says.

View on HN

> "Using fancy tools like neural nets, boosting, and support vector machines without understanding basic statistics is like doing brain surgery before knowing how to use a band-aid."

Having studied both statistics and neural networks, I'm not sure if I completely agree with that quote. There are lots of neural network applications that have little to do with statistics (image recognition with convolutional neural networks for example).

I am pretty sure that the author means neural networks for statistical applications though.

Image classification has everything to do with statistics: you're guessing the probability distribution over the classes conditional on the input image vector; the model is trained through a process of statistical inference (using gradient descent).

You have stated something that someone with a high school-level (e.g. superficial) understanding of probability/statistics would understand. Most research in neural networks requires only a very superficial level of statistics.

There are certainly many areas of neural networks where statistics is important (more theoretical areas), but those don't form the core of the research field.

Also, calling (stochastic) gradient descent a form of statistical inference, while technically correct, is a ridiculous stretching of the term. No researcher considers SGD to be a statistical inference algorithm.

>There are lots of neural network applications that have little to do with statistics (image recognition with convolutional neural networks for example).

You're kidding, right? The most fundamental reasons that deep convnets work at all are statistical in nature.

Well, yes and no. It's like saying you need to understand Bernoulli's priciple before designing the SR-71. True, but 7 levels short of where you're working.

> "Using fancy tools like neural nets, boosting, and support vector machines without understanding basic statistics is like doing brain surgery before knowing how to use a band-aid."

Or like programming in C without also knowing assembly and compiler theory? Or flying a plane without having a degree in aerodynamics?

I think we can extract a lot of use from high level frameworks that abstract away much of the gritty statistics and math. For most applications all we need is to have well behaved, well tested libraries and some basic intuition about how they work.

Fortunately, in machine learning almost everything is a function from inputs X to outputs y, and we know how functions work from programming. It's easy to integrate in apps. The devil is in hyperparameter tuning, but we can get away with good initializations and some measure of web research.

In time people will just use precomputed neural nets for standard tasks like Syntaxnet (text parsing), Inception (image classification) ore use web APIs to hosted services (less secure for sensitive data). We make those better, maybe fine tune them to our needs and get away with it 100x faster.

There is also work in automated hyperparameter search. Machine learning could become a black box when they get good enough.

Straw man going on here. The problem is, whilst you can get numbers out without understanding the statistics underneath, you can easily misinterpret that output if you don't understand it.

Look at how p values are used in science journals for an example of poor stats knowledge affecting real life outputs.

There are some things in life that do require you to do the requisite reading. Things based on stats fall into that camp.

I would agree with you and I also did research on neural networks when I was in college. I'm not sure if those downvoting you have actually done/seen neural network research. Deep learning is pretty engineering driven (as opposed to theory driven), right now.

I have a pretty weak understanding of statistics and from my perspective it was very common for grad students to also have a weak understanding (of course, those working in statistical learning theory had a strong understanding). This is a pretty ordinary occurrence - it shouldn't be surprising - this is sort of like pointing out that many theoretical statisticians have poor coding skills.

> since I wasted a lot of time doing fruitless surgeries before I went and learned about band aids in depth

I'm going to just assume you're being literal here. It's really brightened up my day to imagine the moment such a person discovered you could close up wounds after surgery.

Suggestions for a more rigorous treatment?

My favorites are Casella and Berger: https://books.google.com/books/about/Statistical_Inference.h...

and Hogg, McKean and Craig: https://www.pearsonhighered.com/program/Hogg-Introduction-to...

They are both good books. The first one is more rigorous, but the second one covers more breadth and I think has better exercises.

Anything with measure theory, though you will regret it :)

Here's a free one I found in the event anyone else is curious:

https://services.math.duke.edu/~rtd/PTE/pte.html

stochastic calculus really hurt me. i did well for it, I TA'd for it, i look at it from time to time in my work, but I still find it very difficult. i don't think i'm much good at mathematics, i just try very hard and hope for the best.

On the probability side, I thought Ash's "Probability & Measure Theory" was good for self-study, although some experience with real analysis is definitely a prerequisite. It can be pretty dense at times, but well worth going through.

Reply on news.ycombinator.com