A Cornell Scientist’s Downfall | Hacker News Reader

A Cornell Scientist’s Downfall

wsj.com

70 pointsmelling53 comments

Abusing statistics:

“Mr. Wansink’s fall from grace began with a 2016 blog post in which he blithely confessed to using improper research techniques known as p-hacking and HARKing. P-hacking involves running statistical analyses until they produce a statistically significant result; HARKing stands for “hypothesizing after the results are known.” The post prompted a small group of skeptics to take a hard look at Mr. Wansink’s past scholarship. Their analysis, published in January 2017, turned up an astonishing variety and quantity of errors in his statistical procedures and data.”

Oh, if only we had a system to pass articles that are submitted to scientific journals to some sort of expert committee, that could look at it and see if it was likely to be valid. We could call it, I don't know, "peer checkup", or "expert review", or...oh never mind.

Peer review can't fix broken incentives.

The issue is that peer review never pays off directly. The time spent in reviews mostly goes down a black hole. Editorships are awarded based on publication record, and not quality of reviews.

As a result, more and more, reviews will tend to be shallow and uncritical.

Truth.

>The post prompted a small group of skeptics to take a hard look at Mr. Wansink’s past scholarship

It's funny that this is basically the peer-review version of harking.

You just wrecked my mind

It's also pretty close to the Baysian approach. The difference being that in the Baysian approach one is "permitted" to revise the hypothesis as new data is acquired.

Correction: Bayesian, not Baysian.

I've never heard the term "HARKing" before. This is exactly what my advisor had me do on our experiments that couldn't confirm our initial hypotheses. I knew it was wrong because we set up the experiments in a way to eliminate confounding variables. So any conclusions drawn on data for hypotheses outside of those would be suspect. The experiments we ran were costly in time to do because we were using feedback from students using our learning system over a semester.

I quit working with her because of that, and also because she was a total asshole. I refuse to donate to my university until she's no longer on the faculty. I don't want her benefiting from my hard-earned money.

This is why I believe in Engineering and not the modern "Academia Science".

Totally corrupt, Ive seen it when I talked to professors about getting my PhD. You prove what they want proven.

One is going to have millions of customers using it, the other is going to have a few biased professors review the data.

If you think this is any different from how industry works, you have not spent enough time in industry.

After-the-fact-hypothesis is constantly used to retroactively justify business decisions.

I have 10 years in Industry, testing hypothesis is testing.

Building a product that is bad because 'X person wanted it' is bad Capitalism and will be punished.

Academia doesnt have this same customer focused delivery.

Hmmm...I think it depends on how fast the feedback loop of bad sales is. There are lots of cases where somebody inside a corporation wants the answer to be "X", because that's what they had advocated. It may be worse for the corporation as a whole, but it's better for that guy's career, and if the hit to the corporation is not immediately visible.

He also shamed a post-doc who refused to do it: https://web.archive.org/web/20170312041524/http:/www.brianwa...

"In the meantime, this Turkish woman's resume will always have the five papers below."

Turns out it won't: 2 of the 5 papers have been retracted.

https://www.frontiersin.org/articles/10.3389/fpsyg.2017.0214...

https://bmcnutr.biomedcentral.com/articles/10.1186/s40795-01...

We replace people who are curious about science with people who are good at getting grants.

Iron law of bureaucracy at work.

"If you pay a man a salary for doing research, he and you will want to have something to point to at the end of the year to show that the money has not been wasted. In promising work of the highest class, however, results do not come in this regular fashion, in fact years may pass without any tangible result being obtained, and the position of the paid worker would be very embarrassing and he would naturally take to work on a lower, or at any rate a different plane where he could be sure of getting year by year tangible results which would justify his salary. The position is this: You want one kind of research, but, if you pay a man to do it, it will drive him to research of a different kind. The only thing to do is to pay him for doing something else and give him enough leisure to do research for the love of it."

Quote attributed to J.J. Thompson via a previous HN comment https://news.ycombinator.com/item?id=14022158

That quote is very profound. I know from my own doctoral research how difficult innovation is and how stressful that becomes. I want to also have a personal life outside of just working and what I saw at my university was unhealthy to the extreme. I got my doctorate, but I quit academia. You only get one life to live and I couldn't see why I should work endless hours for shit wages, worrying about obtaining funding year in and year out, dealing with the terrible communists who run US campuses these days, and so on. It was a completely undesirable lifestyle with only a few redeeming qualities.

Bottom line, for me it was less headache to become a software engineer and take longer vacations once in awhile. I would rather do hard research or R&D, but it is what it is.

Honest question: what do you mean by "communists who run US campuses these days"? Is this about bureaucracy, or particular leftist views being upheld, or something else entirely?

You may enjoy the Connes interview linked here, discussing (in the 2nd half) French vs US vs Soviet systems:

http://infoproc.blogspot.com/2018/09/the-french-way-alain-co...

Could someone with more experience expound on why this is the case?

Could you not just alter slightly the desired output? - Here's the 5 things we tried this year, why we failed and why we think it's still worth pursuing

I can trace that to 1971, BACKGROUND FOR MAN, PHYLLIS DOLHINOW - VINCENT M. SARICH

https://books.google.com/books?id=FvDpJ2ulVSUC&q="If+you+pay...

Starting to get mixed feelings about these cases. Impostors and fraudsters certainly need to be exposed. But publicly ripping apart few individuals who, together with their peer reviewers, in some cases may simply not know better (e.g. because their whole field has low standards – nutrition science seems to be one of these fields –; or because they are inexperienced and their co-authors and supervisors don't care about details and correct science and just want to have their names on the paper) can not be the solution for an overall malfunctioning scientific communication and incentive system, or for bad methods that are used by entire fields.

The current situation also invites all sorts of abuse. Need to say that certain parts of the open science movement appear to be a bit angry, not to say sadistic. Wansinks case was one that was ripped apart on Twitter. Wansinks research has received lots of media attention before - would his statistical methods have been investigated and debunked without that? How many Wansinks are out there, still having their job, continuing to apply bad methods (while not committing active fraud), never to be called out for their mistakes? I've been in Japan during the Obokata stem cell case. Her public apology was broadcasted on national TV. In konbinis you could see various tabloid newspapers with her name and her crying face on the cover for weeks. There seemed to be different mechanisms at play than a genuine interest in correct science. One of Obokata's supervisors, Yoshiki Sasai, committed suicide as a consequence of the case.

These people ostracize and hassle those who try to do a honest job and point out the issues with their methods. They also drive the people trying to do a good job out of the field because they can't compete with the fake productivity people like Wansink enjoyed.

Also, the unnecessary suffering caused by this behavior is basically unimaginable. Who knows what advances we would have made if 90+% of resources dedicated to science during this golden era of cheap energy hadn't been wasted? We can't even estimate that.

But yea, Wansink is just the tip of the iceberg. I've been saying for awhile that future generations will need to double check everything since about 1940, because the professional researchers have not been checking each others work, and have been using known incorrect methods to draw conclusions (NHST).

I think in situations like this an idea worth considering is a truth and reconciliation process.

The idea is everyone is given a year or so to go back, identify anything that they may have p-hacked or otherwise been unethical, and if they are radically transparent and work to correct the record then they can continue on reputation and position intact.

It requires some serious compromise, it's imperfect, but it may be the best way to get the truth and prevent an ugly process from dragging on indefinitely.

Also e.g. The Atlantic wrote about this 2 days ago:

https://www.theatlantic.com/health/archive/2018/09/what-is-f...

http://archive.is/II8xf

off topic, but does anyone know how the technology behind archive.is works and how they make money (assuming they do). it seem pretty impressive to me that it always gets the article right.

The site is a commercial enterprise, and as such can go kaputt at any given point, especially if it does not find a lucrative business model. Although it's not a strong indication of long-term issues; in October 2016 the site "made transparent" the server costs, and started to accept donations. A weekly crowdfunded target of $800[4] is set to maintain the site.

Prior to this, the site actively refused donations. A donation link took the user to an animal shelter donation page.

Stated in January 2017, through donations the site only receives "more than $1.50 every day, enough for a bowl of phở".

Src: https://www.archiveteam.org/index.php/Archive.is

I thought archive.is was part of the internet archive project which is a public library of California.

Nope, archive.org is a whole different organization. Also note that the "Wayback Machine" is only one project at the Internet Archive. They also have 78 RPM records, video games, tons of books, CD's, etc. https://en.wikipedia.org/wiki/Lists_of_Internet_Archive%27s_... You can donate at https://archive.org/donate

Edit: Archive Team is also a separate group.

> The site is a commercial enterprise, and as such can go kaputt at any given point, especially if it does not find a lucrative business model.

To be fair, public and nonprofit enterprises can also go kaput at any given point, especially if they don't secure sustainable funding.

also relevant: https://andrewgelman.com/2016/12/15/hark-hark-p-value-heaven...

The guy has a BS in Business Administration, a MA In journalism and a PhD in Marketing. He is a professor of Marketing at Cornell's School of Economics.

In what universe would such a person be called a scientist and not a huckster doing what hucksters do?

I guess he has marketing down to a science.

"Social science". The term science has been appropriated by so many pseudosciences that it has lost any meaning. Because real science has such a good reputation, all the academic frauds have decided to link their field to science to enhance their own reputation. There is political "science", legal "science", economics, etc.

Here is richard feynmann's poignant take on the matter. BTW, it has gotten way worse since feynmann's interview.

https://www.youtube.com/watch?v=tWr39Q9vBgo

Also watching a youtube video on Javascript makes you a "Software Engineer". Bit of a slap to what an actual engineer (in any concentration) has to go through to get a degree.

See also "Harary's Law" <https://quoteinvestigator.com/2015/06/26/not-science/>, which says that any field that has the word “science” in its name is guaranteed thereby not to be a science.

Computer science isn't a real science?

"I've never liked the term "computer science." The main reason I don't like it is that there's no such thing. Computer science is a grab bag of tenuously related areas thrown together by an accident of history, like Yugoslavia. At one end you have people who are really mathematicians, but call what they're doing computer science so they can get DARPA grants. In the middle you have people working on something like the natural history of computers-- studying the behavior of algorithms for routing data through networks, for example. And then at the other extreme you have the hackers, who are trying to write interesting software, and for whom computers are just a medium of expression, as concrete is for architects or paint for painters. It's as if mathematicians, physicists, and architects all had to be in the same department."

Paul Graham http://www.paulgraham.com/hp.html

It isn't a science period. Computer science is mathematics, not science. Alonzo Church and Alan Turing ( the fathers of computer science ) were mathematicians. Algorithms is a mathematical/logical concept.

Theoretical computer science isn't science, it's maths, just like theoretical physics.

But there are plenty of empirical studies in computer science that are certainly science.

> Theoretical computer science isn't science, it's maths, just like theoretical physics.

Yes. Theoretical physics is theoretical. But physics is not. Computer science is purely theoretical. It's math. Adding a theoretical designation to computer science is being redundant.

> But there are plenty of empirical studies in computer science that are certainly science.

Plenty of "empirical studies" that you failed to name one? There are engineering tests for computers and algorithms. But there isn't a "science". What natural law is postulated within computer science?

You can use computer "science" like any other math to do science, but it isn't itself a science. It's a mathematical tool.

Materials science?

He may not be a huckster, just self-deluded -- poorly educated in statistics, math, and the ways of science. It's possible he sincerely believed his so-called "discoveries!"

Regardless, I agree he does not deserve to be called a "scientist." Calling him one is an insult to actual scientists.

Pretty sure he did believe, otherwise he wouldn't have made the infamous blogpost encouraging p-hacking.

Science is a process, not a degree. If you approach your problem scientifically, using the scientific method then you are a scientist, no matter what your degree is or which department you belong to.

Now this person in question almost certainly was huckster, but he would have been just as much of a huckster if his PhD had been physics and he'd been a physics professor.

Eh, paywall.

Install this extension to use Outline.

https://chrome.google.com/webstore/detail/outline-read-witho...

Only works on Chrome, so not an option.

You don't need the extension to get the benefit. Just go to https://outline.com

Please don't post unsubstantive comments here.

Re paywalls: if there's a workaround, it's ok. Users usually post workarounds in the thread.

This is in the FAQ at https://news.ycombinator.com/newsfaq.html and there's more explanation here:

https://news.ycombinator.com/item?id=10178989

https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...

Reply on news.ycombinator.com