Human psychology and behavioral studies overlook 85 percent of people

They could be attractive to the desperately impoverished single adults, who are mostly men.

" [...] This suggests that even under the (likely unwarranted) assumption that the discrepant results are all false positives that have found their way into the syntactic literature due to the shortcomings of traditional methods, the minimum replication rate of these 469 data points is 98%."

98% replicability!! Compare that with psychology.

> Wait, does participating in a study as a subject count for credit, or does working on a study count?

There was this course called "measurements and diagnostics" and for that you had to pass the theory/tutorials of the course and also 10 hours of being a participant in research. If you didn't do the 10 hours of participation, you could do them at any moment you want and if you completed it, you immediately passed the course.

My university has about 400 psychology students, so that is 4000 hours worth of people participating per year. And I'm pretty sure that more universities are doing this. In The Netherlands alone I estimate there to be about 40000 psychology students per year and probably about 5 hours of compulsory 'participation' in experimental studies. So a tiny country like The Netherlands is skewing this particular bias about 20000 participant hours per year.

Participating in experimental studies is quite fun (pro tip: go for the fMRI and EEG studies ASAP! ;-) ), but I didn't appreciate how I was forced to do it. I didn't learn too much from those experiments and I felt used. It furthermore is a way to be lazy regarding participant recruitment. I'm pretty sure that some startup could disrupt this particular problem. If you have ideas about it, email me, I'm willing to brainstorm and help you to look for ways to solve this problem.

> Aren't studies explicit about the demographics of their participants? And don't studies that make generalized claims usually control for things like gender and age?

Studies are explicit about the demographics, which is why the WEIRD article could be written. And yes they do control for things like gender and age and in general they are quite careful making those claims. The issue is though as a research field psychology isn't the study of the human mind and human behavior but of WEIRD people their mind and WEIRD people their behavior.

Like a lot of societal issues that are in popular media today, this is a systemic issue. Though I could imagine psychology researchers being like: "Hmm, I could get a representative group or I could just get some psychology students and churn my next paper out twice as fast!"

I'm use to the alternative being a research paper that can easily take 20 times longer to complete than being a test subject.

Unfortunately, much like with more physically oriented doctors, opinions and practices will vary widely (including going against established research for whatever reason)-- and those seem to be particularly diverse in this field. The replication crisis clearly shows a strong need for reform at the research level as well.

In my opinion, the way we approach these issues should be 99% focused on what we know facilitates better outcomes (short and long term) for actual patients in a cost effective way. Theorizing about fun new methods is exciting, but what we need most these days is more disciplined and well trained pragmatists in the field, who can put their world views aside for a while to provide basic care for the (emotional equivalent of) bleeding gunshot wounds in front of them. How we get there is a big question, but one part of it is always going to be recruiting and properly equipping a certain percentage of incoming students to do the work.

"Of the authors who selected and defined the DSM-IV psychiatric disorders, roughly half have had financial relationships with the pharmaceutical industry at one time"

https://en.m.wikipedia.org/wiki/Diagnostic_and_Statistical_M...

In a normal situation I would probably refuse and ask them to clarify, or challenge their assumptions or something. In this situation I just try to get over my aversion to answer and try to interpret it as best as I can. Sometimes I got no clue what it could mean and choose neutral or whatever. It's mentally exhausting.

This doesn't invalidate the methodology. Just saying I don't like it. If people refuse to answer or answer in a way that doesn't fit the schema and they throw it out, that's obviously a problem though.

Edit: To maybe answer your question, I skimmed through the actual article and they mention things like old people not culturally accepting a young person administering questions, children thinking they should not speak in the presence of elders. But also other weird things as mentioned that seem to me as stemming from confusion. I mean, you get better at answering with practice, so the whole concept might be totally baffling to some people which haven't gone through a Western school, where you're also quite often expected to answer unclear and confusing questions on tests.

More edit: The PhD students and professors designing these questions and the college students used as test subjects are basically the most overschooled people on the planet. They are the ones that, in school, excelled at answering abstract, underspecified, questions with multiple choice answers. Many have seen these before. They might even know how they are made and how the results get processed. Of course they're going to have less trouble answering.

I'm not even sure how you test something like that for reliability.

I think improving these things is difficult, because of you change the questions, you also loose all the accumulated data, and maybe now you might have a better questionnaire, but you wouldn't know it, and you wouldn't know what the results mean. I mean, until you administer it to just as many people as the old one. It's a lot of work.

The amount of armchair criticism of this sort of thing always surprises me.

There's literature on the the number of response options and it basically says:

People respond quicker with binary options.

You lose information though. People get pissed there's no nuance available (no "maybe" for example). They refuse to answer because they can't say anything other than thumbs up or thumbs down.

As you increase the number of options, your ability to predict increases. Predict the same thing later, predict other things, etc. It stabilizes though.

It does take longer for people to respond, but then they complain about what "slightly agree" means.

Regardless, though, you can ask people to do it, and if they just do it, you can predict from it.

Even when you let people omit responses and refuse to respond, you can model that refusal to see what it predicts and then use their refusal anyway.

You can model how people use the response options in different ways, and in general it doesn't matter too much.

You can even eliminate rating scales altogether and use entirely different systems (forced-choice between different statements), but those don't actually work much better either.

Yes, you can always respond in a cheeky, subversive,or manipulative way, but then that's an entirely different issue altogether. You can always do that, and all you've shown is that you're smarter than a rating scale?

As for the original article... these criticisms have been made for decades. Decades. These are my general impressions:

1. The problem of western focus in behavioral sciences is a problem of western focus in the sciences period. Many behavioral scientists would love to do research across multiple sites but cannot afford to do so. And this doesn't stop all sorts of other biomedical research from being done on very narrowly defined groups of people (or animals).

2. Effects observed in undergrads generalize a lot more than the author is letting on, and effects observed in western populations generalize even more. I'm not saying it's not important to study things cross-culturally, only that the idea that people are fundamentally different in different settings is itself flawed. Many things have been examined across different cultural settings, and the differences are not all that dramatically different. In fact, in one recent replicability study, the effect of culture/sociogeographic population was one thing that didn't seem to matter that much. Some studies replicated and others didn't, but the sociogeographic setting didn't seem to matter very much.

I agree that being more sensitive to human variation is critical, but like a lot of things with behavior, there's a lot of grey areas, which people don't like to hear.

There are whole fields within psychology and the behavioral sciences devoted to these issues. People have put a ton of effort into studying them, considering all the issues being raised here as well as many others to numerous to count, and it's like all that work gets brushed aside like snow in the wind because of random blog posts and anecdotal experience.

But yes, all these studies whose the authors authoritatively make claims but these are are based on people answering questions regarding themselves (how often do you cheat on your partner?) - I never believe these numbers.

Edit: to clarify, there are many other questions that will seemingly be somewhat related or unrelated that will infer later on in the test if you were lying. People who aren't familiar with the tests won't notice it in an honest testing.

Effectively, clinical psychology serves apology for societal and economic ills by blaming the effects on the individuals themselves.

The process is strikingly transparent from an objective standpoint, but ironically it’s precisely an objective standpoint that is undermined by the institution. These days, you can visit one and tell them just about anything, and they will diagnose your disposition as illness.

I don’t blame psychologists themselves, and who is to say they don’t help people? Many surely do, but the institution is an absolute drag on our economy and society; something like a collective excuse for avoiding self-critique.

Sociology theoretically steps in here, but it is merely a study, and some branches of it would result in a totalitarian nightmare.

What’s needed is humanist sociology and humanist psychology. As we learned in kindergarten class, there is nothing wrong with us. Each of us are unique and beautiful in our own ways. The questions are about how to live with one another.

Pyschology as a study is another thing, but as a medical malpractice it has proven antithetical to systems design from the start.

It could have something to do with their written language, for example, and shapes similar to triangles and squares in it appearing in a certain pattern that isn't the basic repeating XXOXXOXXO...

Or it could be music and rhythm is a big part of their culture, and XXOXX then follows with XXXO in a large number of their songs.

Those would have been things recognizable to me. Things I liked a lot.

Also, they would have put my mind somewhat at ease, I think.

I wonder how much of the inability to answer some questions is actually down to feeling slightly intimidated by the onslaught of questions, their sterile aura, not knowing the person presenting them, etc...

How much does the subconscious feeling of intimidation influence the ability to think?