Comment by mlthoughts2018 | Hacker News Reader

Comment by mlthoughts2018 | Hacker News Reader

Parent Full thread

This doesn’t make much sense. If you are not already trained in vector calculus, functional analysis and basic classifier and regression algorithms, then most of these reading list items are completely inapplicable (or even dangerous, like when someone reads some blog posts about slapping together neural nets in Keras and suddenly thinks they can build a model suitable for production).

On the other hand if you are trained in ML, most of these are not detailed or extended enough to give you anything useful. Doing “a hacker’s intro to X” over and over really, really doesn’t give you any skills. This is particularly true for deep domains like reinforcement learning, computer vision / image processing, and natural language processing.

Meanwhile, basic design of experiments for A/B testing, explanatory modeling and simple regressions is a fraught area. Not understanding extremely rigorous details about hypothesis testing, model checking, limitations of statistical significance, etc., can lead to wildly incorrect inferences from poor models that non-experts will completely fail to detect.

At best this list seems like “special topics to seem trendy in AI without getting deep / practical insight into any particular area.” There are one or two minor exceptions in the list.

I think, like it or not, the end goal of the AI industry will result in people with less and less rigorous education creating models used in production. It may not be solid science, but it doesn't have to be. It will end up changing entire fields anyways. Especially for less serious applications, no one cares about rigor.

It's just what happens to a field as it becomes more universally accessible.

I don’t think this is true. Security primitives are more widely available in software libraries now than ever before, but you don’t see people believing they can read a “security for hackers” blog post and then roll their own encryption tool or secrets management tool and use it in production. More than ever, the distinction between a security professional and a layperson who read about RSA algorithms is hugely critical.

It’s precisely the same with machine learning and statistical inference.

The lower barrier to entry just means the danger of releasing extremely unsafe projects is much higher, and the careful validation from experts is that much more critical.

If the model produces good observable results in practice it works. For security it‘s different: It may seem to work, yet be completely insecure (security being the goal).

Without statistical expertise, how would you know the model produces good results?

(If you say something like, “just look at the business outcome of the model” you are proving my point about the danger, because that would be a catastrophically bad way to judge the performance of a model. What accuracy metric did you use? Why? Did you understand training / serving skew? Did you look at a confusion matrix or study class imbalance? What about missing data? What statistical test did you use? Did you adjust for multiple-testing? Was there peek-ahead bias? Did you test for discontinuities and non-linearities that can render p-values inapplicable for tests of linear models? What was your training convergence like? What simple baselines did you compare to?)

No, it‘s not. Most machine learning is used for practical applications where it‘s easy to judge if it works.

For example, if I provide some automatic assignment of let‘s say a related object, reducing the work required to find the correct one and the team using it is much faster, then I can measure this and that‘s enough. Likewise for, let‘s say a recommender system. The BI department will look at conversions and that‘s in fact the only thing that matters. Nobody cares how good the model is. What we care about is whether the recommendations have the desired effect in that case.

And besides, what‘s hard about measuring how accurate a model is for predictions? Compared to most things in a typical CS curriculum that‘s rather easy indeed.

Finally, what‘s „dangerous“? ML is usually applied in a business context. Better recommendations; automatic assignment of objects. Automatic classification of pictures. Things like that.

Sure, in a medical context perhaps or financial. Then it may be dangerous. For everything else, maybe you get a worse result. That‘s not unlike everything else in software engineering where many people with different skill levels get very different results (some better some much worse).

Careful with this reasoning!

I have this anecdote of a industrial sorting/production machine that worked for literally a year until someone opened the door on a windy day and everything went flying thanks to the AI.

Thanks so much for reviewing the list. I am trying to get my head around Machine Learning, I faced the same problem you mentioned. I started getting real understanding lately by reading some original papers , studying digital image processing, and reviewing my Math skill.

Please can you give a study list that would be useful for one trying to ML?

Reply on news.ycombinator.com