Hacker News Reader

Comment by aaronharnly | Hacker News Reader

(I'm asking in all earnestness) Are online surveys now an accepted part of the social science repertoire? How do they compare to previous methods such as phone surveys, which I imagine had more uniform distribution to the population but had their own participation bias problems?

View on HN

I'm not a social scientist, but online polls have been widely used and accepted for years. All forms of polling have bias and these can be adjusted for.

Internet-only polls fared well in 538's analysis of 2012 election polling: http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-po...

Internet polls are significantly different from Amazon turk; see my sibling post. Internet polling isn't nearly as problematic, especially for a study like this.

crowd-sourcing platforms need their own bias adjustments and I'm not even convinced that it's possible to adjust for the bias caused by using these platforms, unless you really are interested in putting a lower bound on human ability. Not a statement about the quality of people on Turk, just the amount of effort they often (rightfully!) put into below-developed-world-minimum-wage work...

Yes. Even and especially surveys where the participants are not only compensated for their time, but are also selected from a pool of people who are specifically seeking out such piece-male work (a la Amazon Turk).

There's a big difference b/w posting flyers/ads in target-rich environments and providing a low friction way for people to get involved in the study. But if you do something on Amazon Turk you're basically guaranteeing a very particular type of person, because random guy with a high paying job probably isn't going to have a turk account and won't have the patience to create one just for your survey.

In some cases you can make a good case that this doesn't totally invalidate any sort of generality in the results, but for the most part I throw out papers -- on face -- that proceed in this ways without extended, evidence-based discussion of why the hell I can expect any sort of conclusion except "our result generalizes to a population of people who really need an extra dollar or two (literally) and are looking for pretty much any way to get it."

Not an answer, but additional info: I was curious how they found participants and looked it up in the paper: they posted their surveys on Amazon Mechanical Turk and paid participants 75 cents to complete it.

So all the survey subject members were Mechanical Turk users? That sounds like a terrible representation.

Sadly it's still a thousand times better than the book it's disputing, which has little or no empirical backing.

Bad statistics aren't better than no statistics.

Looking at your comment I find this funny. All you make is an assertion.

If you take a group that's not representative of the population and you know nothing about the underlying distribution - that's not giving you better information it's just letting you be mislead into making statements which are erroneous.

"Social science" isn't a singular amorphous blob, and these methods aren't uniformly accepted.

Online surveys are certainly becoming more popular as they are significantly cheaper to conduct than the alternatives, and yield publishable results that garner media attention. There are peer reviewers that will be sympathetic to these issues, regardless of the method's robustness.

However, there are others that would say this reeks of dredging (p-hacking) in a very murky pool of data. Their "scepticism" rarely makes the New York Times (or a bestselling book), though.

I think the (IMHO valid) point here is that that survey is a whole of a lot better than the anecdotes in the book that prompted them to do the research. It isn't about providing definitive evidence - but about providing better one.

Reply on news.ycombinator.com