Hacker News Reader

Comment by User23 | Hacker News Reader

> I said that if you had reason to believe in a linear relationship, then you could try to find it. That’s the huge catch to linear regression: no matter what data you put in, you’ll always get a “best match” line out.

This challenge generalizes to all model fitting. Incorrectly assuming a distribution is Gaussian is a big one.

View on HN

Another one is handwaving that a distribution is normal when n > 30 because "central limit theorem!"

Amateur statistics is full of magical numbers and thresholds where everything "just works" :)

When I applied to my MSDA program, I had to interview with the lady who would become my first year mentor. "Now, do it over again... on paper." That's what she'd say to anyone too confident in their outputs. She has a reputation for weeding out people who can pass the entrance testing and qualifications, but can't adapt. And what we do requires thick skin. You're wrong until you just happen to be right.

This is what stresses me out about people doing "bootcamp" data science courses and then getting jobs where they're given the responsibility of making decisions based on analysis of data.

Statistics is more involved than it appears, there are so many ways something can subtly violate underlying assumptions, or appear to be fine on the surface while being actually meaningless. It's really easy to fit a model, it's more involved and difficult to actually understand the complexities.

The variable of interest may not have a Gaussian distribution, but its expected value and variance generally are. Sure, there's some pathological cases, but the Cauchy distribution doesn't show up that often.

Does the Cauchy distribution ever actually show up?

Less often than meetings start and end on time.

Yes. Check the example section of:

http://www.math.nagoya-u.ac.jp/~richard/teaching/s2019/Cauch...

If X, Y ~ N(0,1), then X/Y ~ Cauchy(0,1)

(Read: If X and Y are normally distributed random variables with mean 0 and standard deviation 1, then X/Y is equivalent to a Cauchy(0,1) distributed RV. This is a useful equivalence to know if working with standard normal RVs, which is often. E[X/Y] does not exist!)

If X ~ Cauchy(0,1), then X ~ normalized student-t distribution.

(student-t puts the “t” in t-SNE, where it acts as a weighting function on the Euclidean distance between points. UMAP uses a parameterized version of this weight function that is essentially a generalization of the Cauchy distribution: https://jlmelville.github.io/smallvis/umap.html)

Reply on news.ycombinator.com