Hacker News Reader

Comment by 3pt14159 | Hacker News Reader

Unless you work for a company obviously known for their ML the "expertise" out there right now is brutal. People are building recommendation engines without knowing the very, very, very basics like Jaccard indexes, ROC Curves, or topic drift. I've even had to explain type two error to someone working on one of these before.

I agree with your general thrust, and you're right, messy data is often 95% of the problem, but even going through just the Google courses will put people in the top 15% in most cities.

View on HN

I took a machine learning graduate-level course from Andrew Ng himself, and I don't recall learning about Jaccard indexes or topic drift. Maybe your sense of what counts as "very, very, very basic" is skewed toward your own experience. There's a phenomenon known to psychologists where people tend to think that the stuff that they know is very easy and basic, so they conclude that anybody who doesn't know what they know must be uneducated. But then it turns out that the person you think is uneducated knows about a bunch of surprising stuff that you don't. I can't remember the term for this phenomenon, but I often remember it whenever I find myself beginning to judge another person's expertise. This phenomenon is also super relevant to the failings of most technical interviews, in my opinion.

There's a bit of snobbiness in different areas of tech, although there also are in different areas of academia and research. At the end of the day, the most successful people are the ones who wouldn't dismiss a DS who didn't know "Jaccard index" or "the Halting Problem".

Are you referring to the Curse of Knowledge?

https://en.wikipedia.org/wiki/Curse_of_knowledge

What you are describing also sounds a little like the Dunning-Kruger effect:

https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

Oh, wish I knew the name of that phenomenon as well.

You probably can't communicate effectively. If you are describing "Type two error" of course you will get eyes glossing over. A huge problem with research fields is their terse banal labels. Confusion matrix anyone?

Or you can just say "false negative", and every CS major will understand you.

I find people in Math and CS have often very different names for the same type of concepts and they could easy understand each other if they stuck to the more common terms.

In this case, saying: TYPE 2 ERROR, makes you look like you are trying too hard.

It's also extremely confusing because very few people remember type 1 vs 2 but false positive/negative has intuitive understanding.

type ii error is statistics, not mathematics. there is no equivalent concept in CS because type ii error relates specifically to statistical inference and hypothesis testing.

that said, if you are just pointing to a box in a confusion matrix and saying "TYPE II ERROR," you are probably trying too hard.

Eh, but if you've taken a machine learning course, you should have seen the notion of false positive/false negative when you cover any kind of classification technique.

but they're not actually equivalent, in spite of tables like this [0]. type ii error is a false negative result in the context of a test, where you have to understand which hypothesis is which and exactly what you are accepting or rejecting (hypotheses are not always as simple as hotdog/not-hotdog); if your listener doesn't know what statistical tests mean or wasn't following the setup, they have to stop you and ask.

[0] https://en.wikipedia.org/wiki/Type_I_and_type_II_errors#Tabl...

Granted, Type II error and confusion matrices are covered in more basic statistical classes, and are indeed important for hypothesis testing.

I think the point the parent might have been making is that many people (or maybe just me) know "type II error" by the far more self-explanatory name of "false negative".

What's a "type two" error?

I had to google it. It's a false negative.

A "Type 1" error, is a false positive.

Is this like how people overuse the term "orthogonal"?

"Type I" and "Type II" errors are some of the stupidest and most obfuscatory academic terminology ever invented, and (as an academic) I absolutely refuse to make the effort to learn which way round they go. Just call the bloody things what they are: false positives and false negatives. (Getting seriously OT now, but Kahneman does something annoyingly similar with his talk of "System 1" and "System 2" in Thinking Fast and Slow).

As a computer scientist/programmer, there are three numbers, 0, 1, and infinity. If you're going to index your errors by the natural numbers, and you've got Type 1 Errors and Type 2 Errors, my next question is what a Type 3 Error is, and you know what my next question after that is.

Otherwise, please take this wisdom from programmers, who deal with this sort of thing all the time, and use an enumeration, in this case, {False Positive, False Negative} will do just fine.

When you are designing an hypothesis test the term positive and negative are not so clear. For example you can test that mean weight of bags is greater than 5.0 kg or smaller than 5.0 kg, both test are different, and some times you can accept both greater and smaller than 5.0 kg. The philosophy of hypotesis test is not as clear as a standard tests for pregnancy. In other terms,in some cases the H0 hypothesis is symmetric (>= versus =<) and is not clear what a positive result should be, you have to state clearly what is the H0 hypothesis. In a pregnancy test everyone agree than H0 is that you are not pregnant, that is in my HO the semantic difference between Type 1 error and false negative.

What are some synonyms to Kahneman's System 1 and System 2 then? Because Type 1 and 2 errors seem to be completely equivalent to false and positive negatives. I think Kahneman motivates his decision to introduce the terms System 1 and 2 quite well in his book, and I don't know of any direct counterparts.

Jonathan Haidt proposed a similar system in his book "The Happiness Hypothesis". He called it the automatic and controlled sides. The automatic side/system 1 is also what's being described in the book "The Inner Game of Tennis". I would summarize the two sides as the reflexive and the deliberate sides.

If the title of his book is justified, maybe the fast and slow systems?

Ahaha, I agree so much the the Kahneman jibe. How could someone so smart pick names so f*cking dumb?!

So, to put this in human terms.

A false positive or false negative, can be like a pregnancy test.

A false positive, can be where the pregnancy test shows your wife is pregnant, but she is not. And the baby never arrives. Phew, dodged a bullet!

A false negative, can be where the pregnancy test shows your wife is not pregnant, but she really is. And 9 months later, a baby accidentally pops out. Oh crap!

Of course false-positives can also be bad. To use your example "you then spend $10,000 prepping for the baby, but it never comes, what a waste!"

> Jaccard indexes

Funny you mention Jaccard; I was looking up if IoU (Intersection over Union) has any other name known to ML people when I was preparing my self-driving car presentation (IoU is used in semantic segmentation), and found out it is called Jaccard index as well. To my surprise, all ML experts I know knew about IoU but nobody about Jaccard. I guess it might depend on which university you attended?

i think the term is more prominent on the NLP side, via information retrieval (IR) and clustering. i first saw it in IR, and you'd see it in stanford's CS 124 or CS 224N, for example. if the parent is talking about people who are working on a system that has text-understanding component, i can understand their surprise.

For anyone else who wondered what the Jaccard index is, it's also referred to as Intersection over Union.

...and if you haven't come across that either, see https://en.wikipedia.org/wiki/Jaccard_index for details.

Was the project the person was working on having any success? If so, you might be unfairly ignoring their positive contributions and focusing only on this one negative sign that you observed.

My ML model is 99.7% sure you're a gatekeeper. Might be a type 1 error, though.

What's topic drift?

It seems to be a specialized term referring to the change in focus of blogs [0] and online communities [1] over time. This strikes me as a very specialized concept, rather than a generally-important term in machine learning as a whole.

Edit: To add my perspective, with years of industry experience and graduate-level machine learning coursework, I have never before encountered this term.

[0]: https://link.springer.com/chapter/10.1007/978-3-319-16354-3_...

[1]: http://catb.org/jargon/html/T/topic-drift.html

Also seems closely related to the concept of stationarity in time series analysis [https://en.wikipedia.org/wiki/Stationary_process]

maybe he meant concept drift? https://en.wikipedia.org/wiki/Concept_drift

I did, yes.

Reply on news.ycombinator.com