Hacker News Reader

https://forrt.org/reversals/ is a great resource for checking to see if specific effects have been replicated.

Ooh, thanks for sharing this.

I've just gotten around to Kahneman's "Thinking Fast and Slow" (2011) and it seems a LOT of the discussion in the book relies on non-reproduced or even reversed results in your list. I hope the book changes significantly in the second half, as the first half is essentially barren after reading your FORRT link.

Most of Kahneman's own work has held up quite well to replication, and that makes up a large portion of the book. But a lot of the social psych findings he discusses have not fared well to replication attempts.

Failure to replicate doesn't mean the original findings aren't true or useful. A thousand things besides "the original finding is wrong" can happen to cause a failure of replication.

If you limit your understanding to science that is conclusive, you will paralyze yourself. This is the reality of scientific study; there are exceedingly few conclusions, only more questions, confusions, misunderstandings, etc.

When scientists say we don't "know" anything, this is what they're talking about. Nothing replicates, everything seems more confusing and less clear the deeper you dig, and the further you stray from the "hard" sciences that are easy to experiment, the more true this gets. Psychology, economics, etc. are insanely complex and effectively impossible to test the way we think of a physics or chemistry experiment.

On the contrary, failure to replicate should definitely reduce your posterior probability over the original findings being either true OR useful. True findings should replicate. There are complexities - maybe you can't perfectly recreate the original conditions, especially in non-experimental settings - but broadly speaking replication is the right standard to use.

The presumption here is that it was a true replication, and that's pretty damn hard to accomplish in the soft sciences. It matters, just not nearly as much or as conclusively as in something like physics.

If a result is incredibly hard to replicate, that's a good indicator that the effect is either so niche, or so small, that it's not actually telling you any details of the world worth talking about.

If 'priming' only works every wet Tuesday in February, after someone has had a cheeseburger with no sauce and took their Grandma to the bingo, perhaps there's some _actual_ interesting artefact explaining it...or it could just be random noise.

A lot of the studies involved do replicate, just with reduced effects.

But the presumption here is that the same experiment is even being run a second time, and my point is that many studies who have the stated goal of replication do so in ways that I find hard to justify.

So even when a study claims to "fail to replicate", I don't think we should just take that on face value.

“This is the reality of scientific study; there are exceedingly few conclusions, only more questions, confusions, misunderstandings, etc.“

That goes too far. Some science really does follow the scientific method and documents the methodology well-enough to be replicated.

That’s the basis for giving deference to scientists on some topics — they aren’t just experts, they have subjected their ideas to the cold hard scientific method and others have replicated the results.

Somewhere along the line we demoted science to just be another expertise, where you can just walk around in a lab coat and have social status and people aren’t supposed to question you. Throw in some jargon. No, that’s not science.

They can’t have it both ways. If scientists want deference on important topics, they need to follow the scientific method and not extrapolate too far from established results. No excuses like being “paralyzed” by requiring replicable results.

Thanks. I actually met Kahneman when he spoke at Google, and I'm very interested in not citing any results that couldn't be replicated. Particularly Optimism Bias, at the moment.

Looking at that site, though: if I search on "Ariely" I don't get what I would have hoped, which would be a list of his publications that have been debunked. Instead, there are some generalized articles not about him.

Just to be clear, are the "credible" charges that he intentionally fabricated data, or that he received data (from assistants/corporations) that was fabricated, but that he didn't catch? Because he wasn't actually collecting the data directly himself, was he? And there's a world of difference between these two things.

You are correct that there’s a world of difference between those two accusations.

Specific example: In the car insurance paper, which was covered by Data Colada, there was agreement among all the four authors that Ariely himself was the only person to touch the data. That data (IIRC) was completely fabricated.

I’m happy to edit this comment and my original one if those details are wrong. I’ll give Ariely a presumption of innocence which he does not deserve and indeed which I suspect he abuses in research integrity investigations.

(Edit) data colada link: http://datacolada.org/98

At the end under “replies” Dan says himself that he was the person in touch with the insurer. It really beggars belief to imagine that there was someone at the insurer who fabricated a dataset which perfectly tells the BS story of this paper.

Also note that another of the four authors (Gino) will soon be fired from HBS for fabricating data in a series of studies.

>First, it is visually and statistically (p=.84) indistinguishable from a uniform distribution ranging from 0 miles to 50,000 miles [5]. Think about what that means. Between Time 1 and Time 2, just as many people drove 40,000 miles as drove 20,000 as drove 10,000 as drove 1,000 as drove 500 miles, etc. [6]. This is not what real data look like, and we can’t think of a plausible benign explanation for it.

It is just remarkable to see how people with enough knowledge of datascience to perpetrate this fraud for so long would also make errors as rudimentary as producing a faked 'random' data set in a uniform distribution. And with zero cars reporting over 50k miles driven. (edit: and so many others...this is worth reading just to see it all, thanks for linking).

These frauds really are not as smart as they think they are. But I guess selling best-selling books about stuff you made up could make you believe you're a genius.

The downfall of Hwang Woo-Suk, the infamous stem cell con artist, started when a whistleblower posted that two photos of stems cells that were supposed to be at different stages were actually cropped from a single photo. You could match them by dragging them in photoshop.

You would imagine a con artist of his caliber would be more careful than that, but still ...

> You would imagine a con artist of his caliber would be more careful than that, but still

They've been doing it for a long time and it's always worked, either because they've always been this sloppy but everyone else reviewing was sloppier or gave the benefit of the doubt for reasons, or that they've put less and less effort into it over time because it hasn't been necessary.

Also the simpler the fraud is in execution the more likely you might be able to explain it away as a simple mistake or lack of familiarity with certain "tradesperson" tools adjacent to the skills you have framed in latin blackletter on sheepskin.

At least for the insurance case it's crystal clear he fabricated the data himself. He is the only person who touched the data in the research team, by the admission of everyone involved. He is the only person with a motive; why would the insurance company fake data?

It's also clear that the data fabrication happened in Excel (based on styling differences that wouldn't pass through other formats) and the Excel metadata has Ariely as the creator of the file. If the data had been fabricated by the insurance company as an Excel file, they'd show up as the creators. If it had been fabricated by them as a e.g. a CSV and Ariely just imported it to Excel, it wouldn't show the inconsistent styling.

Just to be pedantic (I am not contesting that this individual was involved in malfeasance or not, just that the enumerated proofs are insufficient at best) -

> If the data had been fabricated by the insurance company as an Excel file, they'd show up as the creators.

Digital forensics does not work this way. It is very rare when many digital evidence line up and imply that an actual person did or did not do something in the digital realm. A user account that shows up in an Excel file is absolutely insufficient to tie a person to the keyboard.

> it wouldn't show the inconsistent styling.

This is also inconsistent how digital forensics works. It can imply that something changed, but nothing about who, when, and why. It could have been an incorrectly propagated filter, or coloring, template, or functions.

Yes, of course I'm not going to retype thousands of words of somebody else's investigation[0] into a HN comment. It's a summary.

> This is also inconsistent how digital forensics works. It can imply that something changed, but nothing about who, when, and why. It could have been an incorrectly propagated filter, or coloring, template, or functions.

No, sorry, I don't agree. It shows the "when" of the data fabrication, i.e. after the data was imported to Excel not before. The nature of the styling inconsistencies shows the "why", since it shows that the data was duplicated and one half of the duplicates was manipulated. Half the data was in one font, half in the other, and the two sets of data can be matched with each other. It also shows the "who" in combination with the metadata, due to the chain of reasoning in my previous message, since it shows that the data fabrication happened after the import to Excel and we know Ariely is the person who did that import.

[0] https://datacolada.org/98

Taking a look at the conclusion in the link you provide, it states:

> it is impossible to tell from the data who fabricated it. But because the fourth author has made it clear to us that he was the only author in touch with the insurance company, there are three logical possibilities: the fourth author himself, someone in the fourth author's lab, or someone at the insurance company.

So you seem to be much more confident about it only being one possibility, when the link explicity provides three possibilities.

Yes, the article lists three theoretical possibilities. I expect the datacolada authors are not stating how likely they think each of those theoretical possibilities is, for obvious reasons. I don't feel under such constraints.

One of those logical possibilities can be excluded entirely thanks to everyone involved agreeing that Ariely was the only person in contact with the company and handling the data ingestion. It could not have been anybody else in the lab.

The other theory is beyond absurd. Not only did the insurance company fabricate the data, but they happened to fabricate results that showed the effect Ariely needed for a high profile publication. Theories like your "slacking off data entry clerk" just don't make sense there.

Now, maybe if Ariely could provide even a shred of evidence for this being the data he received, I could change my mind. But he can't produce the file he received from the company. He can't tell who at the company sent it to him. The trail just runs cold at Ariely.

> they happened to fabricate results that showed the effect Ariely needed for a high profile publication

He'd already done lab experiments that showed the effect he thought existed, and it's easy to imagine an associate who actually fabricated the data already knew what he was looking to confirm.

Again, I'm not saying I think he's guilty or not. I'm just saying that from everything I've read so far, it seems plausible it was sloppiness and not checking data for signs of manipulation. That this sloppiness isn't "beyond absurd", but rather more like the classic "never attribute to malice that which is adequately explained by stupidity" (or by an overworked/overambitious academic).

> He'd already done lab experiments that showed the effect he thought existed, and it's easy to imagine the associate at the company knew what he was looking to confirm.

Based on Bazerman's response, actually, he hadn't:

> [The fradulent paper] came together in a merger of two prior non-published empirical efforts. Mazar-Ariely independently provided the data for Study 3, while Shu-Gino-Bazerman had written a paper containing two laboratory experiments (Studies 1 and 2). The Shu-Gino-Bazerman group knew of the Mazar-Ariely data from multiple public presentations by Ariely.

I think reading the authors' responses is illuminating here.

Mazar's response is basically "oh god, I'm ashamed this happened. I played no role in the data collection, and I don't know who did. But thank you for all the hard work you've done in exposing this fraud, and I should have done better at questioning the data."

Gino's response (for context, she has been implicated in other fabricated data studies) is basically "I'm ashamed. I didn't collect the data, and I trusted the people who did collect it. But thank you for all the hard work you've done in exposing this fraud, I should have done better, and this is what I'm doing in the future to do better."

Ariely's response is "I'm the only one of the authors who had the data. Also, it wasn't me."

Bazerman's response is "Thank you for exposing this fraud. Let me explain how the study came to be, and the history of my own doubts about the quality of this study [which includes two episodes which seem to be red flags]. I should have pushed harder on those doubts [he was in the minority who wanted to retract the paper after nonreplication in the follow up]."

Ariely's response to me feels like raising a preemptive defense against being thrown under the bus. Neither Mazar nor Gino implicate anybody (only disclaim responsibility themselves), whereas Bazerman suggests that it would have had to have been Ariely or Mazar. (There's an interesting bit where Bazerman has an unspecified issue with the [now known to be faked] data prepublication, where one coauthor has unconvincing responses but a later coauthor was able to provide plausible explanations, complete with reference to the underlying data file--but both coauthors are unnamed). Given that, I just don't find the other possibilities all that plausible (and in the case of the middle possibility, it's not exactly exonerative anyways).

I've got other people's names and dates in some of my Office documents metadata. Sometimes it's easier to wipe all but the first two pages out and go from there, than to keep track of other people's templates.

Clearly he had control over the file, but I wouldn't put too much weight on the metadata when there's plenty of other evidence.

> why would the insurance company fake data?

It's easy to imagine reasons. A lazy assistant who's supposed to go through 13,488 filled-out forms decides to count 1,000 of them and then fabricate the rest so they can spend the week watching soccer online instead? Or an employee lost the data on which form results were associated with which condition, and they might get fired or not get promoted if caught, so they made it up? And so forth.

And metadata as you've described it doesn't sem like a smoking gun either. It's easy enough to imagine Ariely created an otherwise empty Excel sheet in the column format he wanted them to insert data into, e-mailed it to them, and got it back with their results. Or he just copied their Excel results into a spreadsheet of his own that already had other tabs, or similar.

I'm not saying Ariely didn't do it or did do it, I'm just wondering if there's an actual smoking gun here, or if it's accusations but there isn't enough evidence to actually know.

to some degree it's not relevant -- the data is bunk, and Dan A flogged it hard.

He also was the only one handling it, so if the data is bad then the responsibility falls to him regardless, both as the handler, as well as the head of the investigation. He either deliberately faked it, or got garbage data and didn't realize it when, now, plenty of others have.

In the case linked in the article, Ariely claims to have received the data from a researcher who not only denies doing it but also points out specific things that are implausible about the construction of the study and the description of how the research was done doesn't align with how she'd have done it or normal university policy

Which points towards either him fabricating it himself, or asking somebody else to give him results without going through normal procedures for registering/conducting the experiment and then naming someone else as responsible for it...

I do think there are cases where academics will have unwittingly published falsified or partly false data and sometimes even garnered fame and citations for it x, and no doubt many more where the academics spot issues with the data they've received and bin it, but this doesn't look like one of them.

> he has been credibly accused of fabricating data in at least four separate studies (Ten Commandments, delta dental, Hartford car insurance, and the study which he claimed Rossi ran for him which is discussed at the link)

Minor point here, the article says Rossi ran the Ten Commandments study. Not sure if there's a different study you mean?

If you read the linked email exchange with Rossi, it appears that while she may have collected some data for him, the method and process used does not conform with the details included in the paper.

It seems no one can provide receipts (which is bad! due to reproducibility), and since Ariely's name is on the paper and Rossi's is not, the onus is on him to have kept that documentation. He asked that Rossi at least corroborate the details described in the paper, and she refused to do so saying that it didn't align with her memory or her standard procedure at that time.

“He asked that Rossi at least corroborate… she refused to do so saying that it didn't align with her memory”

Maybe the whole scandal is an elaborate memory priming experiment?

A fair point: there may be three unique articles referred to in my list there, rather than four. An additional fake paper on which I could have added to that list revolves around an apparently-modified paper shredder.

However (!) there are a bunch of other Ariely articles which are extremely suspicious and which are suspected of being fake. At this point I would say everything he has ever written should be assumed to be faked.

I expect a broader investigation will reveal more fraud. Apparently Duke doesn't care enough but hopefully some renewed attention to the case will force them to take action.

> - don’t click or share this stuff when you see it on Reddit, HN or anywhere else.

I think this point is under appreciated. I also want to add another point

- Don't take a (strong) position on something you don't understand or have equally strong evidence for.

We live in a world of specialization, which means a lot of us "are dumb" with respect to a lot of domains. There's nothing wrong with that. But there's a propensity for society to generalize this intelligence, which makes people think that because they can adequately reason about one domain that they can do so in another domain. It is kinda like the Murry Gell-Mann Amnesia, but to ourselves. Look at a domain you _are_ and expert in, and you'll probably realize that a lot of nuance matters, resulting in many things that would be non-intuitive without that high level of detail. Then ask yourself, do you have sufficient nuanced understanding for this other domain topic? I think we know the answer. Opinions are totally fine to have, but it is the strength of them which matters, and places like Reddit and HN are full of people who have been taught that their intelligence generalizes far better than it does, because what we're comparing ourselves to is other people's ability to reason and rationalize and not the distance our knowledge/reasoning is from the truth. i.e. you can out debate someone while being an absolute walnut.

>Don't take a (strong) position on something you don't understand or have equally strong evidence for.

...or else what?

We all live in a world of specialization certainly, but we also live in a world where we must make decisions with imperfect information. This idea of "only take positions on things you fully understand" is a luxury that isn't practical when you realize how frequently (and rapidly) those decisions must be made.

One thing that matters even more than nuance is execution, and a not-so-great idea executed extremely well is much better than a super well researched and understood idea that didn't get executed at all because the person with it took forever getting to that level of understanding.

There is, as in all things, a middle ground. Take positions, make decisions, but accept you'll be wrong decently often, and that's okay.

> ...or else what?

I'm not sure how to answer this without sounding rude. But you'll make a fool of yourself if you do, put people at risk, and probably lose money.

Considering the other context of your post, I'm not quite sure where your disagreement is. I specify _strong_ as a qualifier to indicate that the strength of the opinion/position (your confidence) should be proportional to your knowledge/understanding/evidence. I explicitly state that having opinions is perfectly fine and would go even further to say that it is impossible to be absolutely devoid of them. Human brains just make conclusions.

To clarify my point, I'm saying that being an armchair expert is not helping anyone. Having some armchair expertise is perfectly fine, but arguing from that position (especially passionately) is generally unhelpful and just adds noise. These are the common posts that you see where someone responds with a very simple explanation that is rather incomplete. You'll often see these highly upvoted as well and better answers buried simply due to their length. The laziness hurts us all! It is better to build habits and communal acceptability to just shut up a lot of times, as the problem with information retrieval is often about accessibility not availability; noise decreases accessibility due to the additional information we must sort through. If armchair comments are given (and they can be helpful, especially on vague topics or where no other level of expertise is present), then admission often helps the person receiving the information as well as creates the ability for better communication to determine higher quality information as well as better allow a more informed specialist to append or correct the basic answer.

The problem we see is not the existence of these armchair expert answers but rather that they are done so with high levels of confidence and the authors often feel the need to defend them, including against actual experts. Corrections or additions are not a critique to one's self, just the knowledge that was laid forth (which is often anonymous, so identity/self-value should play a lower role, but doesn't). Being wrong is not just okay, but should be welcomed as nerds we pride ourselves on being "smart" which means we pride ourselves on _gaining_ knowledge. Not everything needs to be a debate and honestly it is in general a terrible way to acquire or disseminate information.

I just want you to understand that you're coming across as if you want people not to discuss most things in their lives, because as you say, we're hardly experts on most things.

Being wrong is completely okay, and trying to police what people discuss with one another is a dark road to go down. Besides, where do you draw the line? Do I need a specific number of course hours completed? Some number of books read? Three or more articles published on the topic?

Discussion, particularly in writing, is a form of thinking, and thinking with others is infinitely more productive than thinking alone, so if what you're asking is for folks to stop thinking about topics they're not experts on, I'm sorry but no. That's not going to happen.

> I just want you to understand that you're coming across as if you want people not to discuss most things in their lives, because as you say, we're hardly experts on most things.

Thank you. If you have any suggestions as to how to make my point clearer, I'd greatly appreciate it. I hope it is clear now, but obviously I can always improve. I do want to promote discussions, but that also means wanting to decrease arguments. Especially arguments that are uninformed and ones which has a clear answer (or lack of) from the domain experts.

Where to draw the line of domain expertise is not clear, you're right. Academia is one example of a way to pass that line but I wouldn't say the only means as it is just a method that correlates with sufficient knowledge gain. But I also wouldn't say that watching some YouTube videos does not constitute expertise (more explicitly, videos by science communicators rather than lectures). The boundary is fuzzy, but that should put pressure to reduce confidence (which is good for science and truth seeking) rather than increase it.

Discussion is not only fine, but actively encouraged. We're having one right now and at least on my side I don't feel like this is a argumentative. I tried to clarify this previously but I guess this is difficult. I'm not sure how to make myself more clear so all I can do is ask that you read more carefully with the updated knowledge that there's a substantial amount that I agree with you on (differences are in finer points) and request that you point out specifically where what I said may be confusing or resulted in misinterpretation. Hoping that this can facilitate a better discussion, if you still wish to have one.

I guess I just don't understand your point, if it isn't meant to stifle conversation in a practical sense. Is there any other way you can put that idea, that might make it easier for me to understand?

As someone working in this space, I largely agree with your take.

I'll only add that compared to, say, traditional economics we don't have a sense of how unique the problem is to social psych/organizational behavior. In the past decade psychology has done a lot more replication attempts and data auditing than have other fields in the social sciences. So it could be that other fields are equally problematic, but we don't know.

This is a fair point. I would like to see more replications AND better incentives around replications.

FWIW... In plenty of economics papers you can't do a meaningful replication b/c there may be no analogous natural experiment. You can only do analysis of the same data. Such "replications" do not always lead to the same results.

The more experimental parts of the econ literature are now very big on pre-registration (esp. in development, e.g.). Not a panacea by any means, but an improvement for sure.

And there are generally higher standards around making code available now. If you work in this area and read the AER you may be aware of a recent retraction in PF due to coding malpractice. The system has improved but is far from perfect.

Yup, completely agree. I think research practices (and incentives for sound research practices) are starting to tilt in the right direction, though perhaps at a slower pace than I'd like.

I'm curious about the mechanical details here. How can a tenured professor at UCLA not be aware that they contributed (not author?!) to a paper cited 3000 times?

So... the mechanical details are that the authors of the paper w/ fake data said that Rossi did the study for them when asked about it.

She wasn't an author, so she had no reason to be involved w/ (for example) the referee process. She was made aware that this was their claim when someone contacted her.

She also works in a slightly different literature IMO. Even if she had read the article, it probably didn't say "the data gathering was done by Prof. Rossi at UCLA"

Thanks!

She's not in the authors list, and a misspelling of her name was thanked in the paper.

>Dan Ariely is a fraud. To be specific: he has been credibly accused

in terms of defamation, I don't think you can summarize "accused" as "is a fraud"

If reproducible results are the concern, the entire field of psychology should be thrown in the trash. Something like 70% of published papers are irreproducible and serve to push an agenda. “Subjective science” in general belongs in the shitter.

What is the agenda?

There’s no set agenda, I generally referring to people that want to validate/cement their opinions as fact.

Seems a bit extreme to throw out a whole set of concepts (nudge, behavioral economics) because one person may have fabricated some data.

How could he have successfully executed this fabrication? Doesn't he work with entire teams of others, don't others review his work, his analysis, the data collection, etc.? The idea that one man, alone, could pull off such a large deception seems, to me, to be pretty farfetched.

Some additional context here is that in academia, there has been a lot of pushback on the burgeoning field of behavioral economics as a result of political viewpoints; nudge is inherently a liberal concept, and puts conservative viewpoints of personal responsibility in a negative light.

Combine that with the ever-present reproducibility problem in all of psychology, and the economists who were made to look like fools (in their view) in the earlier years of the 21st century are striking back with exactly these kinds of arguments (economics isn't strictly a "hard science" insofar as it can't really run experiments the way physics can, so it's easy for economists to point out experimental issues).

Ariely's books aren't "BS", and throwing that in there kind of shows your hand. Large portions of his books are explanations of the work of others, and even if everything this UCLA professor says is true, that doesn't invalidate a single thing he wrote about in his books, other than perhaps specific citations of his own work (of which we're talking about a tiny handful of hundreds of published and peer reviewed studies he worked on with thousands of other people).

My understanding is that other people in this field, such as Amy Cuddy, who popularized "Power Poses", have also been revealed to be frauds.

Not sure how that's relevant, unless you're suggesting that because some frauds exist in a given field, therefore all members of that field are frauds?

That's the way I view astrology.

Astrology is many orders less predictive than psychology...

Huge swaths of behavioral economics have replicated successfully. Care to cite peer-reviewed studies related to astrology that have successfully replicated?

It is not clear to me whether Amy Cuddy was a fraud OR whether she just published a "statistically significant" result without really understanding it or making any attempt to replicate it in a large sample. I lean towards the latter on the basis of reading about her case probably nearly ten years ago.

> Seems a bit extreme to throw out a whole set of concepts (nudge, behavioral economics) because one person may have fabricated some data.

I specifically said don't throw out nudges - some of them work and replicate e.g., retirement savings defaults, while others are BS like the honesty pledge. Real experimental/behavioral economics is not this media-attention-grabbing garbage anyway. As I said, I have friends in the field. They don't do this BS.

> Doesn't he work with entire teams of others, don't others review his work, his analysis, the data collection, etc.?

No, he doesn't. See the Data Colada responses from his three coauthors on the car insurance paper. All admit he was the only author to have data access. If you can find it, check the earlier, separate investigation into his other fabricated paper w/ Delta Dental where the insurer itself said "we didn't give him that data."

> Some additional context here is that in academia, there has been a lot of pushback on the burgeoning field of behavioral economics as a result of political viewpoints; nudge is inherently a liberal concept, and puts conservative viewpoints of personal responsibility in a negative light.

I disagree on many dimensions w/ your characterization of behavioral econ AND the entire point is irrelevant here.

> economics isn't strictly a "hard science" insofar as it can't really run experiments the way physics can, so it's easy for economists to point out experimental issues

I can introduce you to ten experimental economists who would disagree. Economists have also been pioneers in figuring out when non-experimental situations parallel experiments enough that we can make the same inferences.

> Ariely's books aren't "BS"

Please continue enjoying them if you like them, but I don't have any idea how you'd separate fact from fiction. Ariely is not the first researcher in this exact field who has been revealed to be a fraud.

You've got personal responsibility and liberalism completely backwards. Liberalism is predicated on personal responsibility; people can be given liberty because people can be responsible for themselves. Conservativism rejects personal responsibility and thinks people need to be put under the thumb of paternalistic organizations such as the church or government to keep them in line.