Hacker News Reader

Universities are in the business of creating/discovering truth and shipping it. A Positivist review quickly shows things like math are easy to prove true. As you work your ways down to the humanities it becomes increasingly difficult. And competitive.

Shipping truth that isn’t powerful isn’t as attractive to your customers as truth that is. So you’re incentivized to develop truth that is. And you do this by hacking the accepted standards in knowledge pursuit by starting with a conclusion and working backwards. You tell a story based on the evidence that’s useful.

Now it’s “the science” or at the very least “there’s studies” and this is useful to both the customers (NGOs, journals, activists, lobbyists, media, anyone that wants to influence policy) and the university (attracts money, reputation and status) and the people shipping it (tenure, book deals, speaking fees).

It’s not a conspiracy. It’s just simple incentives. The poor guy who spends his time figuring out what the truth is not or that ships truth that isn’t immediately useful to the customers is looking for a new job after his grant dries up.

The other level of problem is that truth in reality means consensus. You can argue about what truth is in reality, but functionally it means consensus.

So it's not even about what is or is not true, or what is recognized as useful in the short term, it's that what defines truth at any given time is what is popularly believed to be true.

This sounds very flaky but it has real consequences because the incentives aren't just to find positive results, it's to find positive results that are likely to be accepted by the majority of the academic field. This kind of positive feedback loop is bound to lead to pressure because science almost by definition has to get some things incorrect — it's how progress is made. So people are actively punished for going against the grain, or exploring areas where our knowledge is limited and therefore prone to lots of negative results because we just don't know.

> The other level of problem is that truth in reality means consensus

That’s the job of the prestigious media. They manufacture consent to the masses. They propagate right-think. Again, not a conspiracy but simply incentives.

Anyways, there is objective truth. Not everything can be settled that way though and you’re right in that “truth” can mean consensus in many instances.

The problem with that is that in many things the dissent is tasked with proving claims untrue, because of the consensus, rather than those making claim to knowledge having to prove it is true. There’s a lot of this today.

Perceived truth is consensus. But greater truth, reality itself, remains true even if not a single person is able to perceive it. You can't actually modify reality with wishful thinking.

For these scientific papers, at a minimum truth==reproducibility.

For humanities and social sciences, it could mean consensus, but in hard science it is different. You are doing experiments that produce data. If someone else repeats the experiment, they should get the same data.

In this case, no one could reproduce the results of the papers in question.

I wonder how widespread this has become. How many scientists, driven to publish, know that it is too expensive to reproduce their experiment?

Exactly. In other words, there is a huge difference between science as "pursuit of knowledge through thoroughly applied scientific method" and science as "product of academic process".

Incentive structures make sure that the latter is dominant in published science.

Even if there was no intentional fudging of data, results and interpretations by scientist teams, the results of science as a whole could still be heavily (over time) influenced by simple biases in selection of grant applications. For example if you only fund researches of type "thing X good" and dismiss all applications trying to prove "thing X bad", you will eventually gather enough evidence for X to appear good, regardless of how objectively "good" it is.

I (and others before and after me) filed complaints that my advisor at UT Austin was falsifying data. University ignored me and gave him tenure. It is commonplace and admin doesn't care as long as it doesn't affect the endowment.

Are you willing to say who it is?

Yeah I'm also curious

> There was at least one year's Tour de France in which all but one participant were later found to have been doping. In other words, you didn't get to that level of that competitive activity, if you didn't cheat, because it wasn't possible to outperform people with such a significant advantage.

This is not true. Even during the 1904 Tour de France where 9 people were disqualifed because of, among other actions, illegal use of cars or trains [1] - 27 riders finished the race.

Tour de France in the modern era has up to 180+ competitors lining up, and there hasn't been a case of 100+ riders being disqualified for doping.

[1] https://en.wikipedia.org/wiki/1904_Tour_de_France#Disqualifi...

Yeah not the whole peleton, but most of Armstrong’s main rivals right? Like Contador etc. (I know his test was contentious).

Indeed, the claim needs to be adjusted to be correct. One of the worst recent years was 2005: https://en.wikipedia.org/wiki/Doping_at_the_Tour_de_France#2...

There, the claim is that of the top 10 finishers, all but one participant were either stripped of result in that race or failed tests/were sanctioned for doping in another race.

(The remaining participant, originally #8 Cadel Evans, was a documented client of Michele Ferrari.)

A one-off consultation prior to his road racing career to see if he the right kind of fitness for road. I don't think the association should be used to tarnish Cadel without further evidence.

Thank you for the correction, I either heard wrong back then, or misremembered. I think the analogy/logic still applies, though. Actually, this suggests even more strongly that the higher their position in a competitive field, the more likely they are doping...

I don't think this kind of fraud is even remotely close in scale to the cycling doping you mention.

My primary reason for believing this is that as a scientist myself, I've never done it, and none of my coauthors has ever proposed to do something like that. I have seen other kinds of fraud (e.g. ghost authorship) but not result manipulation.

But of course, If I were a fraudster myself I would also say that I've never done it :)

So a more general argument: scientific "doping" is just not as straightforward as cycling doping. In cycling, it's clear that if you don't get caught (and have no scruples) you benefit from doping, it makes you stronger.

In science, you first need an idea that looks original/promising, and then you do the experiments and see if the idea works or not. This kind of "doping" will only help if you (1) are good at generating ideas, but (2) these ideas often don't work. At least in my field, this is possible but far from the most common situation to be. And in other situations, there is not much benefit to faking results.

> if you (1) are good at generating ideas

I have been joking with my former boss (who is a professor) about how easy it is to come up with good ideas. We each have five to six One Billion Dollar ideas before breakfast, and another 1-2 while in the shower! /s

Coming up with ideas is _ridicolously_ easy, executing on them is the hard part. If you're a seasoned researcher, it shouldn't be hard to guess at the "meta" of your field, you're probably aware of promising ideas discussed among your colleagues. What if you faked results, proving one of these ideas?

And the bar for cheating is _suprisingly_ low. I'm not a researcher myself, but I've experienced how my work is torn apart in peer review. But if you're dealing with data, apparently no one makes sure to actually check it before approval.[0][1]

[0] https://youtu.be/d2Tm3Yx4HWI?t=846 (analysis on the original data from the Francesca Gino scandal, which already looks suspicious without statistical analysis)

[1] https://youtu.be/KsSuhP60qnI (faked data from the Hendrik Schoen scandal at Bell Labs, as long as you make outrageous claims no one can verify, you can continue to publish in top journals)

I suppose this is heavily discipline-specific. In my specific subfield of CS, the execution is often a Python program or bunch of scripts that can be (and often are) coded by an undergrad if given enough guidance on what exactly is wanted. And then of course the way of presenting it in the paper, choosing which data to show, etc. but cheating doesn't help much there. The hard part in my view is coming up with what Python program should be written (i.e. coming up with a relevant algorithm, experiment, evaluation methodology, etc.).

I can see how in other fields, e.g. biomedical, execution could be more challenging and ideas more plentiful.

Anyway, I don't think ideas (read: good ones, because obviously anyone can have nonsense ideas) can be a commodity in any field. It is quite widely accepted that, e.g., when applying for an ERC grant (the highest-regarded individual research grants in Europe), having a good idea is as important as the CV if not more. This wouldn't be the case if coming up with good ideas were so trivial.

> having a good idea is as important as the CV if not more

I don't know about the acceptance criteria for ERC grants, but from the peer reviews I've read from one of my professors papers it feels like there's a bias towards accepting novelty of all else. My joke with ideas was meant more general, i.e. that you can have a few billion dollar ideas, but executing on them is much harder (due to lack of funds, time or skill).

Additionally, depending on the demand of the field I can imagine it being easier to obtain funding for resarch. Imagine being in machine learning right now lol.

I wasn't talking about grant acceptance though, but purely what kind of ideas can get you published and what data you have to fake in order to get there. In the case of Francesca Gino it doesn't sound like her fraudulent resarch was so costly that she would have to have applied for large grants in order to conduct it.

You make it sound so straightforward and simple.

> and then you do the experiments and see if the idea works or not

The problem is "if the idea works or not" can be very fuzzy. With a little p-hacking, anything can work! If nothing in your original data/experiment supports your hypothesis, you can just cherry pick some part of it that gives "significant" results.

>My primary reason for believing this is that as a scientist myself, I've never done it, and none of my coauthors has ever proposed to do something like that.

Are you sure? Also a) how far along are you and b) how far from actual research work are you at this point?

I'm a full professor and PI of several projects. So sadly more management than actual hands-on research lately, but I've gone through all stages, of course.

And yes, I have more than 100 papers and in their making I haven't fabricated data or seen anyone do so. If there is fake data in any of them, it would be because someone included it behind my back. But I doubt it's the case, as during the course of my career we tried various things that didn't work and we just moved on to other experiments. I've never had a student or collaborator even suggest "let's just edit the data and publish it". The most that we have done is to try to cast negative results in a positive light to make them publishable ("the results are promising...", "the results are mixed but show improvements for some dataset..."), but without touching the data or saying anything factually incorrect.

Doping in sports is so much more complicated than most people think it is.

For starters there’s so many performance enhancing substances, and entire underground industries for developing and producing them. All of these drugs would be banned by the blanket prohibition on unapproved substances, but in practice many of them don’t have any screening procedures, so won’t turn up in any tests.

There’s also the fact that many performance enhancing substances are biologically identical to substances your body is naturally producing (like bio-identical testosterone for example). The only way to test for these is to test the ratios of certain metabolites (with a margin for error to avoid false positives). In practice this basically means any athlete can do steroid cycles, as long as they are only boosting their test levels up to that of a genetically elite outlier.

You also have the therapeutic use exemption, which allows for legitimate therapeutic use of banned substances. Testosterone replacement therapy is a legitimate therapeutic use of steroids, and some sports are known for abusing TUEs for this usecase.

Testing for PEDs is way less precise than a lot of people think it is, and any motivated athlete can get away with PED use to some extent, no matter how much testing you subject them to. There’s good reason to be suspicious of a lot of athletes. One massively suspect category of athlete is all of the ones in/near their 40s who somehow manage to maintain physiques as good as/better than they had in their early 20s.

I would be very much interested in reading more about this, do you have any good articles you could point me to?

This paper is a bit out of date, but gives a reasonable overview of the challenges of testing for testosterone supplementation

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2657495/

Paper on challenges of detecting growth hormone

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3830297/

Testing for both of those substances has improved, but still leave plenty of room for abuse.

Use of "unknown" or the constant stream newly developed substances is naturally not rigorously studied, but here's a couple of articles referencing the challenges they pose.

https://thetruthaboutforensicscience.com/undetectable-perfor...

https://bleacherreport.com/articles/1955231-undetectable-the...

UFC is notorious for testosterone TUEs, but that's mostly because they regulate things differently to most other sports

https://www.espn.com/espn/otl/story/_/id/10500652/therapeuti...

Top flight soccer is another sport that's constantly garnering attention for TUEs, but most of the coverage of that is speculative, because other than high level statistics the TUE process is naturally not very transparent (and most of the recent coverage on the topic has been replaced by controversy relating to trans athletes).

In my personal, anecdotal experience...

Once you get far enough in ANY sport, you start considering tradeoffs between risking long term health and improving performance. If you are a competent adult, you can evaluate your drivers, consider risks.

But without fail, in ANY competitive sport, you realise that you have to risk health in order to achieve that next performance tier, there is no other way.

Once you get into that 1-10%(depending on sport) elite level, you realise that there is no way you can compete without PEDs.

I've participated in (taken myself) and have been a witness to this in weightlifting(olympic and strongman), soccer, cycling and swimming. Starting young. Real young.

Coaches of different sanity levels are trusted to make the right choices and not often questioned by busy or absent parents.

Everyone cheers on when medals are won and records are broken, without consideration for how it happens.

I personally had no idea I was taking hormone modifiers until about 16, it started at 11.

My coach just gave me what he said were vitamins and I never questioned it, he was the closest thing to a father I had, his authority was absolute.

At 16 I found myself in a hospital yellow as a banana on the brink of liver failure, that's when I found out.

Everyone on our team took what the coaches gave. Once a while we'd see a parent get into a shouting match with one of the coaches about something we didn't understand and that kid would never come back again.

Some of the guys I trained with went on to become highly recognisable names in their respective sports. One even has a supplement brand. Those I've kept in touch with continue PED use, that is the price of being at the top of the game.

Depending on the substance and quality, the time it is detectable can be as short as 12 hours to a few days. There is a reason lots of athletes refer to drug screenings as "IQ tests". Only the dumb or careless get caught.

You've hit the nail head on. Also applies to China's anti-corruption programs. Everyone who achieves any level of political power is corrupt, but framing it in an anti-corruption campaign allows you to target your political enemies.

No need to single out China. All politics, and guess for Academia, can extend to all human organizations, Church, School, Government.

You’re making the preposterous assertion that all countries’ politicians are equally corrupt. That needs some extraordinary data to support.

Didn't say equal. Just that all 'large' human organizations tend towards corruption.

Churches, Governments, Academia. Hierarchies form, incentives drift over time and become mismatched with original intent. The proof is in history, look at Church abuse, Inquisition, look at grade fixing, admissions scandals, look at Volkswagen tweaking emissions, look at CEO's lying to promote products. Our entire society is based on the ebb and flow of some kind of corruption, or ebb-flow of Law.

Even today, in America, half the population think that the laws themselves are corrupt and should be broken. Judges aren't trusted. So who is corrupt if you don't have any foundation to judge anything against?

I'm totally against China taking Taiwan, but from a 'certain point of view', they have point. It was a part of China before the war, and the mainland 'won'. So they think of it as unification.

Again. I am totally against China taking Taiwan. Just think it is bit more nuanced.

Actually, think we are agreeing.

If there’s no need to ‘single out’ China then there’s no need to ‘single out’ doping in cycling, and no need to ‘single out’ this one particular Stanford President researcher, etc. The objective of your comment, that I am pointing out, is that you are hand-waving corruption in Chinese politics away by saying “but what about…”

I agree with that. If we hand-wave away all cases, then we never tamp them down. Corruption would continue to grow.

So, we should call out all cases. Even if there are so many cases to cause one to become tired of the fight.

Maybe I was just reacting that in todays world the lines have blurred enough that it is sometimes hard to tell if you are fighting corruption, or if you have become the corruption.

From different points of view both these are correct:

"The US experienced a Coup attempt by a president that then did not suffer any consequences for his un-constitutional actions"

or

"The President, fighting for the American Way against the corrupt deep state government attempted to bypass a fake election to continue his quest to root out evil? "

> It actually became a contest, not just of bicycling, but of doping, because getting away with it was quite difficult and it took years for the world of cycling to get good enough countermeasures to shift the advantage to non-doping.

Not just a contest of hiding doping, but of winning against others who were doping. If everyone is doping, then you could argue it's still a fair playing field, just with a higher skill ceiling.

It’s not a fair playing field because there were still people who did not cheat.

That's why I said, "If everyone is doping" :)

But that's not what you were actually implying.

As long as they knew others were doping, I would argue it was still a fair playing field. The real advantage is in others not knowing you are illegally enhancing performance.

https://en.m.wikipedia.org/wiki/Christophe_Bassons

> If everyone is doping, then you could argue it's still a fair playing field, just with a higher skill ceiling.

No, because now you have at least two contests in parallel: the race event for the cyclist and through-the-year "doping and not getting caught" for medical/management team.

For example one prominent olympic level swimmer was disqualified because they failed to appear for testing multiple times. Why would one do that just to come back after disqualification?

There are at least two problems with this argument. First, faking a research in a way that won't be detected is hard, especially if aim for high impact paper which probably will be studied by others. That's why we hear a lot about fraud in top tier venues. Lower ones probably have more problematic publications. Second, having fake results won't guarantee you a high quality publication, leave alone position in the future.

I can imagine some kind of first order phase transition in which the amount of fraudsters becomes so large that self correction breaks and instead they start to endorse each other. But that'd be evident from the outside, since science will stop producing real life results. We're definitely not there yet.

It doesn't have to be outright fraud. I imagine the vast majority of manipulations are extremely subtle. Intentional selection bias, multiple hypothesis testing without correction or reporting the failed hypotheses(p hacking). Most of these would improve your chances of finding important results significantly and be almost impossible to detect

For the first argument I can't really speak beyond my field (physics, or specifically condensed matter physics), where I think it's fairly hard to be subtle. Often it's partial data presentation which allows for inverse Occam razor [1] (proposing fancy explaination instead of a simple one), which is why full data publication requirement are important (and seem to happen more).

In smaller sub (sub)fields sometimes fraud becomes large enough to self-sustain, like it happened with Majoranas [2,3] but even that kind of self-corrected. [1] https://arxiv.org/abs/2204.08284 [2] https://twitter.com/spinespresso/status/1503352928656138241 [3] https://espressospin.org/2022/11/17/the-fallen-angel-particl...

> without reporting the failed hypotheses(p hacking).

This one is so subtle, it's probable that many of the people doing it don't even realize they're doing something wrong.

> First, faking a research in a way that won't be detected is hard, especially if aim for high impact paper which probably will be studied by others.

Fake is a spectrum. Soft faking is easy and widespread--"p-hacking" is the polite term.

The stakes for faking it are low because most research is low impact, most research is seldom reproduced, and even bad research is rarely retracted. For that which is reproduced there is plausible deniability--"researcher degrees of freedom". Getting a high impact study published is more important than having one retracted, so researchers follow that incentive structure.

> But that'd be evident from the outside, since science will stop producing real life results. We're definitely not there yet.

Have you taken a look at the biosciences, social sciences, and medicine? Basic research hasn't been reliable for decades.

> The stakes for faking it are low because most research is low impact ... Getting a high impact study published

That's contradictory. For low impact studies you can fake whatever you want, because nobody really cares. That probably wouldn't promote your career either tho. Publishing bunch of organic semiconductor papers in Nature and Science will promote your career, but won't hold. That's the point.

> Have you taken a look at the biosciences, social sciences, and medicine? Basic research hasn't been reliable for decades.

Not deeply, but I surely can come up with bio/med research with real life applications that appeared in last 2-3 decades. Say mRNA vaccines?

The entire fields of sociology, social psychology, economics, literature, theological studies, etc are all low impact. Even if you get in the best journals, it’s high impact for your career but low impact to society so it will get very little scrutiny apart from the tiny slice of academia you compete with.

Again, I know that academia is very different among fields, but usually high impact for career ~ high impact for (sub)field, meaning people will want to look at your data and possibly try to replicate it. Outside scrutiny for science is extremely rare anyway

> That's contradictory.

What is contradictory in a claim that most research is low impact thus easier to fake while the incentives for publishing high impact research skews to publishing bad/fake papers since publishing is far more important then having a paper retracted?

> Not deeply, but I surely can come up with bio/med research with real life applications that appeared in last 2-3 decades. Say mRNA vaccines?

Nobody said that all research is faked.

Ok, not contradictory, it is just two claims unrelated both between themselves and to my original claim. Yes, low impact research is easy to fake. Yes, there is incentive to publish fake high-impact papers. Yes, it is hard to create and publish fake high-impact impact research.

It doesn't even need to be deliberately faked data but more data published without much due process to ensure it validity.

I have had about three people working in the research fields that have used the term, off the record of course, "Publish or perish".

Get it out the door, try to get the funding and hopefully if it works out, solve the issues before it becomes a problem.

I suppose you could probably also say the same of certain housing markets. If the cost to rent or buy a place is sufficiently high and the system actively prevents the prices ever coming down to a point where people who didn't inherit huge sums or came into that money some other way have a chance, you might as well do whatever you can because the constraints have been laid before you. At a certain point, you're just evening the playing field and surviving.

I know several Tour de France aficionados, who would claim that doping remains rampant, they are just better at hiding it now.

Any competition without an effective referee (from the Tour to Academics to presidential races) will devolve to the lowest level. People always seem surprised by this...