huitzitziltzin

A fair point: there may be three unique articles referred to in my list there, rather than four. An additional fake paper on which I could have added to that list revolves around an apparently-modified paper shredder.

However (!) there are a bunch of other Ariely articles which are extremely suspicious and which are suspected of being fake. At this point I would say everything he has ever written should be assumed to be faked.

I expect a broader investigation will reveal more fraud. Apparently Duke doesn't care enough but hopefully some renewed attention to the case will force them to take action.

On the contrary, failure to replicate should definitely reduce your posterior probability over the original findings being either true OR useful. True findings should replicate. There are complexities - maybe you can't perfectly recreate the original conditions, especially in non-experimental settings - but broadly speaking replication is the right standard to use.

This is a fair point. I would like to see more replications AND better incentives around replications.

FWIW... In plenty of economics papers you can't do a meaningful replication b/c there may be no analogous natural experiment. You can only do analysis of the same data. Such "replications" do not always lead to the same results.

The more experimental parts of the econ literature are now very big on pre-registration (esp. in development, e.g.). Not a panacea by any means, but an improvement for sure.

And there are generally higher standards around making code available now. If you work in this area and read the AER you may be aware of a recent retraction in PF due to coding malpractice. The system has improved but is far from perfect.

So... the mechanical details are that the authors of the paper w/ fake data said that Rossi did the study for them when asked about it.

She wasn't an author, so she had no reason to be involved w/ (for example) the referee process. She was made aware that this was their claim when someone contacted her.

She also works in a slightly different literature IMO. Even if she had read the article, it probably didn't say "the data gathering was done by Prof. Rossi at UCLA"

It is not clear to me whether Amy Cuddy was a fraud OR whether she just published a "statistically significant" result without really understanding it or making any attempt to replicate it in a large sample. I lean towards the latter on the basis of reading about her case probably nearly ten years ago.

> Seems a bit extreme to throw out a whole set of concepts (nudge, behavioral economics) because one person may have fabricated some data.

I specifically said don't throw out nudges - some of them work and replicate e.g., retirement savings defaults, while others are BS like the honesty pledge. Real experimental/behavioral economics is not this media-attention-grabbing garbage anyway. As I said, I have friends in the field. They don't do this BS.

> Doesn't he work with entire teams of others, don't others review his work, his analysis, the data collection, etc.?

No, he doesn't. See the Data Colada responses from his three coauthors on the car insurance paper. All admit he was the only author to have data access. If you can find it, check the earlier, separate investigation into his other fabricated paper w/ Delta Dental where the insurer itself said "we didn't give him that data."

> Some additional context here is that in academia, there has been a lot of pushback on the burgeoning field of behavioral economics as a result of political viewpoints; nudge is inherently a liberal concept, and puts conservative viewpoints of personal responsibility in a negative light.

I disagree on many dimensions w/ your characterization of behavioral econ AND the entire point is irrelevant here.

> economics isn't strictly a "hard science" insofar as it can't really run experiments the way physics can, so it's easy for economists to point out experimental issues

I can introduce you to ten experimental economists who would disagree. Economists have also been pioneers in figuring out when non-experimental situations parallel experiments enough that we can make the same inferences.

> Ariely's books aren't "BS"

Please continue enjoying them if you like them, but I don't have any idea how you'd separate fact from fiction. Ariely is not the first researcher in this exact field who has been revealed to be a fraud.

You are correct that there’s a world of difference between those two accusations.

Specific example: In the car insurance paper, which was covered by Data Colada, there was agreement among all the four authors that Ariely himself was the only person to touch the data. That data (IIRC) was completely fabricated.

I’m happy to edit this comment and my original one if those details are wrong. I’ll give Ariely a presumption of innocence which he does not deserve and indeed which I suspect he abuses in research integrity investigations.

(Edit) data colada link: http://datacolada.org/98

At the end under “replies” Dan says himself that he was the person in touch with the insurer. It really beggars belief to imagine that there was someone at the insurer who fabricated a dataset which perfectly tells the BS story of this paper.

Also note that another of the four authors (Gino) will soon be fired from HBS for fabricating data in a series of studies.

Summary for those who are not deep into academia…

Dan Ariely is a fraud. To be specific: he has been credibly accused of fabricating data in at least four separate studies (Ten Commandments, delta dental, Hartford car insurance, and the study which he claimed Rossi ran for him which is discussed at the link). There are likely more.

More broadly: There was a “golden age” for this kind of “behavioral economics”/social psychology work in the early 2000’s up to probably 2012-2015.

Researchers would generally publish “results” which were either “delightfully counterintuitive” (ex: prime subjects to think about Florida and they walk slowly) OR featured extremely cheap interventions which showed large effects on outcomes (ex: signing an honest pledge at the top or the bottom of the page made a big difference in how honest people actually were on their tasks).

In retrospect, there are two problems (at least!) with this literature.

1. There was no theory backing up any of these predictions, so you could rationalize any statistically significant result with a compelling story. Publish it and you’re on your way to tenure.

2. Related to (1.), you didn’t even need to bother running actual experiments at all! Just fabricate the data. Make it fit your cute story and you’re done.

There are many tenured people in business schools (generally in “organizational behavior” or “social psychology” style departments rather than Econ) who were tenured on the basis of papers published under (1.) or (2.).

This is a big shame because there are serious researchers who do difficult work in behavioral and experimental economics which is _not at all_ like this BS. (Some are personal friends of mine, so this bothers me.).

There was no incentive for about 15 years to do the hard work though. And given that tenure is for life, it’ll be a long long time before these people are flushed out (barring more investigations, which should be encouraged!).

What should you do?

- be extremely, extremely skeptical of any results in the “priming” literature (eg the “think of Florida” example above).

- be moderately skeptical of results in the “nudges” literature. The retirement savings stuff makes sense and has replicated but the honesty nudge above is BS and has not.

- don’t get really excited about a researcher who says “economics bad - people dumb! - I behavioralist!” More than a few of these people have been revealed to be total frauds.

- don’t click or share this stuff when you see it on Reddit, HN or anywhere else. That kind of engagement “feeds the beast” - Deans love it and Deans assume you haven’t fabricated your data.

- maybe if you are specifically a Duke or Duke Fuqua alum send them a note asking why they aren’t taking the Ariely misconduct more seriously???

- (edit) throw away your copies of any of Ariely’s books. They are BS.

Some of the inferential steps in this article are… “heroic” might be a polite way to put it.

“Furthermore, the fact that none of the infant burials found in Valencina have grave goods suggests that, among the communities that lived in or frequented this site, social status was not ascribed by birth (and therefore no significant inheritance of wealth occurred), contrary to what would happen later in the Early Bronze Age)32. Therefore, it is fair to assume that the individual inhumed in structure 10.049 gained a prominent social position through merit and personal achievement and did not ‘inherit’ it by birth.”

I mean… on the basis of the tombs that have been found you want to infer that? Sorry but there’s ten other stories that fit the available evidence. This interpretation really stretches credulity beyond the breaking point.

The strangest thing to me is that American education professors wield power in setting educational standards, when their own field is completely allergic to any serious empirical work on what works. I guess to a state legislator a “professor of mathematics education” sounds like the kind of person who should know about these things?

California math standards resemble New York reading standards in their lack of engagement with studies of what works. There are too many examples like this. We would all be better off if education professors stayed completely out of the business of setting state standards.

How about “cross swords in court” at the end there?

Truly amazing an amazing title in the original too.

There was a previous “too good to be true” post from this same sub stack author a few months ago.

IIRC he claimed that zoning was nearly going to end due to the builder’s remedy provision in some recent CA bills.

When I searched for other commentary from informed CA lawyers it seemed like the claims in the article were exaggerated.

I hope that these are not exaggerated bc the changes outlined would be a good step, but do keep the caveat in mind.

Don’t miss that the effect on athletes is quite a bit larger than the effect on legacies! See page 4.

Legacy admissions are indefensible, but athletic admissions are even less so. (Less than 0, I guess?)

No one outside of Harvard cares how their fencing team does and (I presume) 99% of current Harvard students don’t either.

If you want to find the group that is most unjustly benefitted by a dumb feature of the admissions process it’s in my view athletes.

The book “neural networks and pattern recognition” by bishop dates to 1996 and has a red cover, at least in its current softcover iteration.

The random forest guy you mean is/was Leo Breiman. His student Adele Cutler deserves some of the credit there too.

The point I am trying to make in 2(a) (maybe inexpertly) is that all the data sharing in the world isn’t going to solve any problems when the people looking at to do their own “replications” are uninformed about methods. Data sharing on its own will not solve this.

>>What prevents it from becoming the norm for the "statistically insignificant" data to be published alongside the rest?

The incentives of the researcher. You don’t want to be the guy whose lab produces ten failures and one success, even if that’s the reality of every lab.

The guy who doesn’t share the null results will look better than the one who does every time. Deans and chairs only know how to count one thing in tenure decisions - lines on your CV.

To put this another way, nothing stops this kind of sharing right now, and yet we have a well known file drawer problem.

If the proposed solution involves “oh let’s just completely change the social norms, despite the incentives of the individuals involved,” then it’s a non starter. The kind of sharing being proposed is that kind of idea.

>>Point 1 is really the one I don't understand. Is it medical/health privacy that prevents this? Surely no one expects unanonymized data to be provided. This is about teasing out bad research, not indicting anyone for the falsified sort.

What you consider “anonymized” is generally not what regulatory agencies will consider anonymized and the best sources for health data at scale are always governments. (Frequently they are the only sources.) The agencies are too risk averse about the potential harms from data releases, but that’s the reality. You can’t share it. You can’t even have it on your own laptop, encrypted or not.

Interesting data that lets you answer complicated questions cannot be made public. The more interesting and the more complicated, the harder the underlying data is to access and if you say you want to share it publicly they’ll laugh you out of the room. You’ll never make it through the already excessively burdensome IRB process.

1. You simply can’t do this in many fields, including wide swathes of health research. It’s a non-starter.

2a. Even if you could, see the OP’s point about research design. You need to understand the methods.

2b. Even if you could part (b), the raw doesn’t help you with publication bias. Your selection of raw data is going to be heavily biased in favor of the data for the papers which managed to be published. It’s not going to contain the data for the results which never saw the light of day due to failure to reach statistical significance.

The response from regulators should be: Let VCs waste their money.

This is an extremely weak analysis from an economics point of view. The regulator has several options to deal with monopolists if and when they emerge.

If consumers realize benefits in the meantime in the form of cheap products subsidized by VC money that’s a good thing.

The UK’s analysis was highly speculative and not that persuasive to me as an antitrust economist.

I don’t think this is likely a good merger for consumers due to the possibility that Microsoft would foreclose PlayStation users. But the EU’s reasoning in terms of “damage to future competition in a sector which might exist but currently doesn’t” was really unpersuasive.

I don’t know UK competition law that well but if they can appeal and the law is like US law then I would imagine they have a good chance at getting the merger approved in the UK.

I’m a little confused by your skepticism. The underlying research is studying what happens given that an interaction reaches a human. Whatever deflection happens before that is irrelevant to the interaction with people afterwards. The paper studies the latter and the results are specific to that set of interactions, which are presumably more complex or ill-suited to deflection.

Congratulations you’ve identified exactly the problem which concerns literally every economist who has ever worked on this question. Education Econ people work very hard to get around exactly your worry.

Education is not my field but there are ways to identify causal effects outside of experiments.

So: you are wrong when you say “the only way to do this is…”

Because if we did that then the missing revenue that schools currently get from their endowments would instead come from…

1.) thin air

2.) the government (??? No.)

3.) higher tuition -> hint: pick this option!

The better question is why Harvard and Princeton charge tuition at all! (The answer to that question is not that hard either.)

Impossibly, unbelievably stupid. On the level of the Bible Code. The Mayans were not visited by aliens. This is so dumb. What is this doing on this site???

Economically, this outcome does not require imperfect information. Dispersion in workers’ marginal product of labor is sufficient. In this case workers may be heterogeneous in the value they place on the time - that may be observed by the firm. Imperfect information might play a role here but I doubt it’s a big one.

Software engineers are also paid differently by company and within companies. I know you know this.

They give you different raises according to a highly imperfect measure of your productivity and value to the firm. Two people with identical tenure and identical titles likely have different salaries on average.

None of this should be remotely surprising. And what’s the alternative ???

Actually I think paying them the same amount is crazy! But I’m glad for the opportunity to clarify in case that was unclear in the original comment.

Paying employees differently for similar work is pervasive and can be justified in several ways.

Calling it “cartoonishly evil” as a commenter did above is ridiculous.

If you don’t like this case, why not mandate a single uniform salary within or across firms?

Really looking forward to the inevitable comment from someone who will try to make a case for that wild idea.

Easily one of the least surprising research findings I have ever read. This is completely normal in every firm. Places where everyone is paid the same despite their differences (eg public schools where the PE teacher is paid the same as the math teacher) are the exception.

Nothing in the piece makes a case for that at all.

The agent says Putin lives in a bit of an information bubble but also repeatedly says he’s solely responsible for the war.

If this guy is a plant… then his FSB handlers really screwed up!

Not a good criticism of this paper in particular.

Read the details of the “save to win” program in the Data section. Save to win was not a conventional lottery program. (Two more conventional lottery programs are also included for which your criticism is more accurate.)

This study covers men and women who got wealth shocks through three different programs, two of which are fairly conventional lotteries and the last of which is a “save to win” program where your odds of winning are a function of your bank account balance (to encourage savings).