How many studies like this need to not replicate before we stop treating them seriously?
> This left us with a final sample of 84 dyads, randomly assigned to either the Scarcity (n=42) or Control (n=42) conditions. Dyads across conditions did not differ in age, child or caregiver sex, caregiver education, or family income
> The caregiver and child were seated across from one another at a table. The child completed an unrelated experiment with the researcher, while the caregiver completed the Scarcity or Control manipulation survey on a tablet. When the caregiver had finished, the researcher left the room under the guise of loading a second survey onto the iPad, leaving the caregiver and child alone with a toy the experimenter happened to offer as she left. A video camera and/or tape recorder recorded their interactions.
And there was also a second experiment:
> The second experiment used [..] tiny “talk pedometer” devices worn by children that record their conversations and count the words they hear and say.
> [..] analyses revealed that parents engaged in fewer conversational turns with their children at the month’s end, a time that typically coincides with money being tight
Hard to answer this, IMO. It would be hard to stop keep versions of such anecdata from informing our personal worldviews, for example. Business knowledge is basically made out of such stuff.
Replicability, for questions like this may be the wrong bar. Should we expect a relationship between financial hardship and talking to three year olds to be the same for a group of parents in 2021 california and 1997 Tokyo?
A lot of the replication crisis is shoddy work. Bad research. But... a lot of it is a problem of the subject matter itself. It's complex, and can't be isolated. Does dumping a ton of manure in a forest result in more leaves? more flies? Hard to describe the relationship between the manure and leaves in a replicable way. You can do controlled experiments, but the lab version doesn't answer the same question... the one about complexity.
IDK... I'm not excusing the replication crisis and many of the worst affected fields do need to pay attention to the gunk they've accumulated. OTOH, we do need ways of describing relationships that are complex. "We know nothing" is insufficient.
"Data Availability: For Study 1, all materials and analysis scripts, including the protocol to be used in the case of a future replication, are available on OSF..."
But, inspecting the data closer, it does seem like the effect is not huge, and that is perhaps why the Berkeley website that is summarizing the results did not choose to mention anything about the size of the effect.
Jokin aside I've seen way better studies of this kind where the parents would wear a mic at home and they would then do a word count, word analysis, etc.
Especially considering that going through a tough time will colour your memory and memory is extremely unreliable.
Paper doesn't really support the headline.
Statistical significance is a measure of the likelihood of getting that effect size by random if the two approaches didn't actually affect the outcome. With a large enough sample size, an effect size of 1 fewer word could be "statistically significant", even if it is practically meaningless.
In this case they got a p-value of 0.1 iirc. So assuming there's nothing special about the circumstances, you'd expect to get samples this different 1 in 10 times you ran the experiment by luck alone.
The authors listed many results, and noted whether each was significant. It seems weird that you keyed in on one measure (caregiver vocabulary) with a higher p-value, and not the other results that did suggest that when money is tight, parents talk less.
I'd still ignore it though. You see how the first 3 are under "pre registered analyses" and the significant finding is under "exploratory analyses"? That's another way of saying their experiment failed to find anything interesting so they combed the rest of the dataset to try and find something they could publish. Basically just classic p-hacking. Probably not malicious or intentionally deceiving, but p-hacking nonetheless.
If you throw your hands up and decide to check what's interesting on a 14x14 correlation matrix, you've just tested around 100 hypotheses without realizing it. If your significance threshold is 0.05, you should expect to have 5 false positives in there already.
Exploratory analysis is not in inherently p-hacking. Publishing exploratory analyses as such is the proper action. There’s nothing in their methodology that suggests they analyzed a large number of possibilities and discarded the high p-values. (How would that even work in this case? Run through all possible conversation topics? All possible time divisions of speech?) The exploratory topics extend naturally from their preregistered hypotheses, from speech and income to speech relating to income and speech on calendar days when income is an issue.
Your cynical take on publishing negative results is unhelpful, as the accusation of bad faith and the straw man.
I'm open to being proven wrong but I've reread this section a couple times and I'm pretty sure all four tests in the primary pre-registered section and both tests in the secondary pre registered section are all non-significant.
(looks like there's actually two studies here. I've only read the first)
> Exploratory analysis is not in inherently p-hacking. Publishing exploratory analyses as such is the proper action. There’s nothing in their methodology that suggests they analyzed a large number of possibilities and discarded the high p-values.
Exploratory analysis is p-hacking. p-hacking is not (always) an evil unprincipled scientist trying to push a story. It's usually a scientist without a lot of statistical knowledge trying to see what's interesting and then letting their personal biases confirm coincidences as they appear because they want to find SOMETHING. You can publish these results, but you'd better be very clear that, you know, it's not very good. They're relatively conservative as they should be in the scheme of things here, and that's good. But look at the article and the discussion its generated. Clearly some people think its a trustworthy "scientific" result.
> How would that even work in this case? Run through all possible conversation topics? All possible time divisions of speech?
You take your data, you generate a correlation matrix of everything you've got, and point your finger at the values that look high. Then you test those hot spots and find significance. Very easy. You're implying here that in order to p-hack you need to check out every possibility. Not true.
They already tested ~6 ideas. Assuming they're independent of each other (to be fair, they're not), the likelihood of finding something significant with a p value below .05, purely by chance is already 0.95^6 => 26%. That is to say, naively, if you run 4 of these studies, one of them is expected to get a positive result even if they're all bogus. If they're allowed to do exploratory tests, they're now getting the freedom to cherry pick additional options. Remember, you have to count both the things they actually test AND the things they decided not to test after seeing the correlation matrix.
If they consider an additional 3 ideas, your studies now have a 33% false positive rate. At 15 ideas you're now at a 66% false positive rate. At about 40 extra ideas you have a 90% chance of finding something significant. If everyone's doing this, then your entire field is probably bogus.
It is extremely easy to unintentionally cheat and write off 40 insignificant ideas glancing at a correlation matrix. Pre-registering ideas is critical.
and I'm harping on correlation matrices because that's probably a lot of people's first idea, but there's plenty of other ways. That is the whole point of exploratory analysis.
You can see for example that one of their exploratory analyses is what happens to the relationships when we group by income? Well, you get to test another handful of ideas, I can guarantee you that much.
> The exploratory topics extend naturally from their preregistered hypotheses, from speech and income to speech relating to income and speech on calendar days when income is an issue.
That's not remarkable at all. That's the kinds of information that existed in their dataset.
> Your cynical take on publishing negative results is unhelpful, as the accusation of bad faith and the straw man
Well I did say "Probably not malicious or intentionally deceiving" so if anyone's pushing a bad faith interpretation here I'd say it's you.
It literally is not.
At the limit, this is "my parents couldn't afford proper care for a debilitating diesease" vs they could. There is one group there that would talk more.
if you're poor, broke, alcholic, meth head, on heroin, you're probably not gonna be talking to your kids as much as someone who is not.
To see that idea with a self survey is kinda hilarious. kutgw to get this stuff right.
Also stop calling Americans "interesting." Using a term like this is a deliberate insult on a culture or a person. You are examining the culture like it's a lab rat and commenting on how the behavior is "interesting."
You are not an idiot. People do not talk like this in real life by remarking on how behavior is "interesting" to the subjects face. You and others only use these terms behind the anonymity of a forum. Therefore you are aware this is insulting. Stop.
This is a common tactic used to get around the HN rules. You say "interesting" posing it as an innocent remark. It is not, you are conducting a deliberate and insulting attack on American culture here.
They're generally not referred to that way outside discussion of behaviors addicts engage in to get their fix and the discussion is limited in scope to addicts.
Making the generalization that everyone refers to the poors as meth heads serves the same ideological needs as generalizing the poors as addicts albeit to a different end.
It wasn't a self survey.
> This left us with a final sample of 84 dyads, randomly assigned to either the Scarcity (n=42) or Control (n=42) conditions. Dyads across conditions did not differ in age, child or caregiver sex, caregiver education, or family income
> The caregiver and child were seated across from one another at a table. The child completed an unrelated experiment with the researcher, while the caregiver completed the Scarcity or Control manipulation survey on a tablet. When the caregiver had finished, the researcher left the room under the guise of loading a second survey onto the iPad, leaving the caregiver and child alone with a toy the experimenter happened to offer as she left. A video camera and/or tape recorder recorded their interactions.
> The caregiver and child were seated across from one another at a table. The child completed an unrelated experiment with the researcher, while the caregiver completed the Scarcity or Control manipulation survey on a tablet. When the caregiver had finished, the researcher left the room under the guise of loading a second survey onto the iPad, leaving the caregiver and child alone with a toy the experimenter happened to offer as she left. A video camera and/or tape recorder recorded their interactions.
The conclusion could be that being reminded or queried about hardships has a chilling effect
To be specific:
> They recruited 206 volunteers—a mix of psychologists and economists, students and professors, none of whom were involved in the SSRP itself
The point is that it's possible to relibly predict whether a study will reproduce based solely on the characteristics of the study.
(I believe one of the early controversies that sparked the reproducability crisis was the discovery that an excess of studies were using college students as their participants, skewing their results)
I'd pay that. Though in this case it's not people reading the paper, but reading the article about the paper. (On reading the paper, the findings are less conclusive than the click-baity headline, but that's almost always the case)
This may or may not be the case or may be true in some specific circumstances (Redskins, for example) but may not be true for all, however, many high schools are changing their team names based on this one study/conclusion which was funded by a tribe. Which is fine, but usually when there is a conflict of interest one wants to have additional independent studies that come to a similar conclusion.
Changing a name in response to a conflicted and un-valid paper is a downside in itself.
In one you change the high school mascot from a native american cartoon to an animal or something. I'm assuming native americans were not consulted with the creation of the mascot to begin with.
In the other we change... what all the teams names? Is that a equivilent scenario?
To be honest I'm finding it hard to relate to someone would care enough about a high school mascot to even need a study at all. Low stakes B.S. But that's me.
Since this topic is probably of zero importance to anyone not native american who else is going to fund it, for that matter?
Seems like you are affirming the consequent and assuming it is harmful to native Americans.
You are ignoring the issue of the paper. If the decision is based on a scientific paper, but the paper is conflicted or unvalidated, that is harmful because of the misuse of the scientific paper.
If you want to change the mascot because of Native American’s preferences, that is a different matter.
If not it's kind of a catch 22 for the tribe, isn't it? Nobody wants to fund research on the topic, so when they fund research on it it's presumed invalid because of bias?
How does that address the harm done by basing decisions on bad science?
At least not until the reactionary media's outrage machine decides to make it their current hill to die on...
Also, this is America, where sports are taken far too seriously.
And somehow you think that schools that have nothing to do with those people finally changing their very questionable names is due to a “conflict of interest of a psychologist”? Not because it was just racist?
Should Aztecas Stadium change their name? What about the MN Vikings, NY Yankees, TX Cowboys? Kaiser Chiefs of ZA, Ebusa dwarfs of ZA?
It could also be argued that it dilutes the image/culture/history and probably a bunch more things by wrapping it up in american plastic. I wouldn't argue with that.
Broadly speaking, the symbolism being used is usually either an obvious caricature of the real thing (see: the former Washington Redskins), or uses cultural and religious elements that are at best kind of offensive in the context of being used as a sportsball logo (see: most uses of peace pipe or war bonnet imagery).
There are exceptions, but they're of the kind that generally come with either somebody actually doing the research or people of the given group actually participating locally.
As someone else mentioned most of the push against is not from American Indians/Native Americans but vicariously offended others.
I've been following the efforts to have team names changed for quite some time and have never heard of this paper, but have heard of sustained lobbying over decades from a variety of native organizations.
Let's rephrase this.
Many high schools are changing their team names because the study brought awareness to the generally ill-considered use of ethnic mascots.
The study may have been a catalyst, but the appropriation and the insensitivity to it was always a problem.
In 2016, when the main momentum of this started to come out, 9 in 10 Native Americans did not find it offensive [1]. But white people had already decided it was offensive, and white people have much more cultural influence than Native Americans, so in being perceived by white people as offensive it became offensive, to the point that just a few years later, a majority of Native Americans found it offensive [2].
But at all times through this, including today, a higher percentage of white people find the Redskins name offensive than of Native Americans. The Redskins changed their name because of pressure from white people, not because of pressure from Native Americans. More than a little ironic that even in this, Native Americans were barely listened to, while white people's feelings were considered very important.
[1] https://www.washingtonpost.com/local/new-poll-finds-9-in-10-...
[2] https://news.berkeley.edu/2020/02/04/native-mascots-survey/
And, to be clear, I'm okay with presenting evidence one way or another, but I object to willy-nilly claims and crusades.
The ethics considerations of microphones always on would need to be extremely stringent, and only for safety of the children or others should they be breached.
The study might also require the provisioning of basics like Internet access and such devices. Though these days Internet access is as necessary for a productive member of society as power and water are, and IMO it should be a universal right.
and would be done by Facebook or Google to sell the data that you need money to loan companies.
No it’s not. I have an uncle who is a plumber and gets by without Internet just fine. Banks with ATMs, branch offices, and phones. Pays bills using the the mail, etc.
In the US utilities are required to offer mail options for bills, but not email or phone. Unless you work from home over the Internet or have children remote learning, it’s absolutely just a luxury.
Plugging your ears and crying ethical foul impoverishes the discussion and all it’s observers.
You could instead add real arguments to the debate. That enriches the discussion.
Don’t get mad at me for calling that person out. I’m the one who wants more research - they apparently already know the answer.