Comment by somenameforme | Hacker News Reader

Comment by somenameforme | Hacker News Reader

There is a severe demographic problem with this study (among others). A current trend in the social sciences is using Amazon Mechanical Turk and Prolific to run surveys. These sites are based around paying people pennies to compete menial tasks (like complete surveys), and populated with large numbers of "professional" menial task executers. This study offered the big bucks of $1.50 - $2.25. There are two major problems with this:

1) It doesn't matter how you normalize a sample when that sample is a niche subsample of society to begin with. This has always been a problem with the social sciences which previously relied heavily on undergraduates in social science classes, but that is going to at least be much closer to something like a representative sample than Amazon Mechanical Turk or Prolific can offer.

2) The people carrying out the tasks are motivated primarily by blazing through as many studies as possible while doing the bare minimum to get paid, and many fail at even that. In this study ctrl+f for "attention check." To this study's credit, they at least included numerous attention checks, but it emphasizes the issue. On average about 1 out of 6 participants were excluded for failing to correctly answer questions along the lines of "Please select C as the answer to this question."

Journals are likely indulging this because it's going to generate endless "impact" (which is increasingly becoming a euphemism for clickbait), but it's undermining the entire field more than ever before.

I wonder if you could get good survey data by disguising it as a Buzzfeed quiz on Facebook. Just make the title something like "Which Kardashian are you?", then ask whatever questions you want and randomly select a Kardashian at the end.

"You are Gul Dukat."

Cardassian, Kardashian, all the same to me!

‘Keeping up with the Cardassians’

I would seriously question my life choices.

I suspect that's exactly what most of those "quizzes" actually are.

I believe that's what Cambridge Analytica did.

> quiz on Facebook

The current FB audience is already comprised of those that are over 35-40 years of age. As a early 40s guy myself I couldn't care less who the Kardashians are (I know the Kim lady, there was also a younger lady that did a Pepsi commercial and some former man that transitioned into a lady, Jenner I think?), but, ignoring that (let's say you could target me with sports-related quizzes), the fact still remains that Facebook the platform doesn't target the entire age-spectrum anymore (if it ever did).

I don't think anyone under 35 years old cares who the Kardashians are either. I honestly can't figure out who does beyond people who are responsible for deciding what stories appear in various places.

As someone who has never taken, and will probably never take, a Buzzfeed quiz: I think it probably risks the same kind of sampling bias. Not sure what the best way to overcome this is. Joke: government mandated surveys with monetary punishments based on your net worth if you fail to take them.

Unless you make survey taking a society wide, randomly sampled, civic duty -- like jury duty -- you're always going to be vulnerable to some biases.

You'd still get only a sub-niche of the population, just a different one -- the one that fills out nonsense quizes on facebook.

I saw another study on hn [1] that "showed that people found accountants and data analysts boring". However, this was all based on forms filled in by workers on MTurk and another crowd sourcing company...

[1] https://news.ycombinator.com/item?id=30764152

Maybe if American society valued science the same we we value justice, we could compel people to occasionally participate in surveys and studies the way we compel them to participate in jury duty.

We see the latter as a fact of life and occasional annoyance for a cause greater than ourselves. Maybe we should treat a contribution to science the same way. Many countries require some military service of young people—we could ask people to contribute in all sorts of ways.

Maybe if social science was real science instead of quackery people would want to help.

If we didn't already have mandatory jury duty, I don't think there's a hope in hell that you could get a bill passed requiring it today. If you did manage to get a bill passed, I think the courts would almost certainly strike it down.

We have become much less keen on requiring people to work against their will. Jury duty is grandfathered in, but new things of that kind wouldn't be lightly accepted.

This is true of a lot of things. Imagine if libraries didn't exist and someone got the bright idea for one today. It would never fly.

In a world of open source and Wikipedia, that's a bold claim.

Those aren't funded by taxpayers like libraries are.

I don't know of anyone that considers jury duty an "annoyance for a cause greater than ourselves". Normal people know intuitively that it is a waste of time, that lawyers, cops, clerks, and judges are just running an extended grift on working people and the rest of society, and that nobody will be harmed by their lying their way out of having to miss work for 7 dollars a day.

What's the alternative? Professional jurors?

In great majority of countries there are no jurors, just judges.

I would say the opposite: shanghai citizens into serving as cops, prosecutors, judges, bailiffs, etc., so that there is some hope that they might influence the process instead of just being another material that the professionals process (like defendants).

Now you do. I just served jury duty this week, in fact, and believe it to be an important (and, yes, annoying) way of serving my country. I find your take disappointingly cynical.

I wrote about this problem a few years ago. Even more relevant today: https://www.psychstudio.com/articles/research-participation

Why would undergraduates be more representative of society than Mechanical Turk users? Surely the Turk users span a much wider range of age and geolocation.

There's also the perennial problem that polls are not public opinion.

The biases within polls are legion and difficult to eliminate in any society with privacy provisions.

Sorry, but the public social sciences are dead anyway. If you want to do research on humanity you go were the actual data is. FGN.

Everything else is peanuts.

What is FGN?

I think it's "Facebook/Google/Netflix."

That makes sense. Big Tech in general has data on billions.

NSA.

Assume you have the conversation of billions. Its pretty easy to design "Virtual experiments", that just play out or have played out as life plays out.

Select from Everyone

//Filter the Environment with constraint to avoid polution

where(Person.isAlcoholic and Person.Age.inRange(12,49))

//Define experimental condition

experimentOutcome(State.Change(Person.isAlcoholic) upon Person.Graph.Depth(1).Death)

Gets you all acoholics quitting after death of a nearby relative.

That way, you get almost pure data and archieve what neuro sciences and psychology could never dream off. To really know people better then they even ever knew themselves.

So sad, as this is "steering" technology, we will never get to see it. We do not even get to contemplate its existence.

It's at least a small step up from when many studies of human behaviour were done on psychology freshmen.

It always amazes me how such studies are simply left up to the academics, because clearly taking representative samples of the population will need some kind of help from the government to organise it

Probably just requires a bigger budget than most academics have. I got paid £50 to spend half an hour giving feedback on a new website, in a 1:1 conversation with a researcher. Granted the funding was from government, but Kantar did the organisation side.

I think it's not possible to do ethically or practically, because only some sorts of people will willingly subject themselves to the testing, examination and prodding of of social sciences, even when bribed. And if you tried to force it on them using a government, which would be unethical, you still have the practical problem of the data you get from people under duress being garbage.

There are many companies that specialize on this. Usually, it takes picking random people at many different streets, but does not require government intervention.

No, all you need to do is hire a professional polling company. They also have biases but they're far closer to providing representative samples than MT is. Academia doesn't do this because it's expensive and they have a cultural bias towards quantity over quality.

The need for representative sampling is something understood by first year undergraduates on first exposure to the concept.

What you've written is the reddit equivalent of criticizing a study with "correlation does not imply causation".

Yeah, scientists are familiar with the idea, believe it or not. Scientists in fact go to great lengths to ensure their sampling is representative, and studies published in good journals are carefully scrutinized for non-representative sampling.

If you can't conceive of how a person working at Mechanical Turk is overwhelmingly likely to just be a normal person, and not your weird conception of them, I'd encourage you to read Thinking Fast and Slow.

Contempt for scientists is not a good look.

I think his point is that it's impossible to normalise such a sample. For example if I sampled only toddlers about their attitudes to ice cream then there's no way to extrapolate from that sample to the general population, and it's the same thing for "people who are willing to fill in a survey for $2".

Understanding the need for representative samples is not the same as actually getting representative samples. It's not a problem of scientists being ignorant or lazy or even underfunded. They know what they need, and they simply cannot get it. They try to make do anyway, but the quality of their work still suffers because of these factors which are out of their control.

Not all fields of inquiry are equally tractable to the scientific method, so they should not be treated with equal reverence.

Without getting into a snark, that is not a particularly good book to recommend.

https://slate.com/technology/2016/12/kahneman-and-tversky-re...

"Taken all together, the chapters Schimmack looked at earned an average grade of C-minus."

Thanks. I had the impression this book and his work was widely regarded. I'll have to put a mental asterisk next to some of the stories I was blown away by.

The book not so much (despite its initial critical acclaim on continuing commercial success), but at least his research summarized within seems to be reproducible and his reputation is intact. The research on priming in particular, was reliant on low sample size studies, and was not reproducible.

Without getting into snark, your very own article bolsters my point:

> The replication crisis in psychology does not extend to every line of inquiry, and just a portion of the work described in Thinking, Fast and Slow has been cast in shadows. Kahneman and Tversky’s own research, for example, turns out to be resilient. Large-scale efforts to recreate their classic findings have so far been successful. One bias they discovered—people’s tendency to overvalue the first piece of information that they get, in what is known as the “anchoring effect”—not only passed a replication test, but turned out to be much stronger than Kahneman and Tversky thought.

What portion of the scientists in the book are represented by Kahneman and Tversky? The other scientists in the book did not go to great lengths to avoid sampling bias. The authors could not be relied on to spot it either.

I don't see any point you made being bolstered. The anchoring effect you mention in another comment doesn't permit a scientist or statistician to throw their hands up in the air and say one mechanical turk group is as average as any other pseudo random group.

Even if you grant the idea that a person working at MTurk is just a normal person, which personally I doubt. You have to contend with both the fact that they're highly likely to be lying about their identity in order to maximize the number of tasks they can be included in, and they're also likely giving trash answers in order to complete the survey as fast as possible. I spent a bit of time doing mturk for a while, and it was absolutely one of the first things I started to optimize - "How do I get through this survey quicker". It's incredibly difficult to earn minimum wage doing mturk, which brings us back to the idea that it's unlikely it's just an average normal person doing it.

It's absolutely possible to produce a rigorous statistical result from the data, but that's useless if the data is all lies.

> Even if you grant the idea that a person working at MTurk is just a normal person,

Just want to point out that surveys are not looking for "normal" participants. They need a representative sample which, given human diversity, would include many people who are "abnormal" across a wide array variables.

Do you have any data or evidence to support your claim that workers on Mechanical Turk are a representative sample of Americans? This is an extraordinary claim, given the incentives of the platform. Think like a hacker, or a poor person in India or Nigeria.

Anchoring effect. Read the section in Thinking Fast and Slow on the anchoring effect.

Reply on news.ycombinator.com