Hacker News Reader

Comment by johan_larson | Hacker News Reader

It's not just p-values. Some people just don't understand even very basic statistics.

I remember talking to one person in marketing who ran surveys of the company's users. They would send out a survey to all registered users, get back responses from 1% of them or something, and then proceed to report findings based on the responses. They were really happy, since a 1% response rate is great for surveys like this.

I tried to explain to them that all of this statistical machinery relies on having a random sample, and a self-selected sample is not that. No effect whatsoever. Surveys like this are standard practice in the industry. Why are you making trouble, geek-boy?

View on HN

" Surveys like this are standard practice in the industry."

So, how does one exploit this apparent bad practice? I.e., how does one make a profit from others making this mistake? The answer is, of course, one doesn't - otherwise others would've done so. So what does this tell us about the state of affairs? Is it that bad sampling doesn't matter for practical purposes, or that marketing research is useless? I don't know, and I'm not trying to be belligerent here. A situation similar to this shows up in dozens of places every day - more often than not, it doesn't matter if things are done 'right', there are large margins within which 'good enough' is indistinguishable from 'right'.

This bothers me greatly, but it's hard to argue against this conclusion, empirically. How to deal with this cognitive dissonance? I mean this is the exact topic at least half of the blog posts that make it to the HN front page deal are fundamentally about.

Had an indergrad ecology course to teach me this. The first 4 labs were plug and chug equations sampling a square with computer generated randomly placed dots. All we had to do is get a sample (we could choose the size), and figure out a 95% likelihood that the correct number was within our range. No one could do it for 3 labs, before he told us how. Good samples are REALLY important, even under basically perfect conditions.

Reply on news.ycombinator.com