Hacker News Reader

Comment by lacker | Hacker News Reader

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well.

It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results. It's common to use things like manual Jupyter notebooks, unlabeled CSVs, and a bunch of disorganized data files, in a one-off process that a scientist manually summarizes to produce a paper.

To me, in an ideal world each paper would sit in a GitHub repository, with an integration test that verifies the code actually produces the results used in the paper. That isn't really what academics prioritize, but perhaps things will move in this direction as more people realize that we have a replication crisis, and also as scientists tend to have more software engineering skills over time.

View on HN

The current scientific system has long been known to have serious problems of incorrect results and conflicts of interest. This article seems like an attempt to pin the crisis on an AI scapegoat.

From 2015, the Editor of The Lancet:

The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. As one participant put it, “poor methods get results”. The Academy of Medical Sciences, Medical Research Council, and Biotechnology and Biological Sciences Research Council have now put their reputational weight behind an investigation into these questionable research practices. The apparent endemicity of bad research behaviour is alarming. In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world. Or they retrofit hypotheses to fit their data. Journal editors deserve their fair share of criticism too. We aid and abet the worst behaviours. Our acquiescence to the impact factor fuels an unhealthy competition to win a place in a select few journals. Our love of “significance” pollutes the literature with many a statistical fairy-tale. We reject important confirmations. Journals are not the only miscreants. Universities are in a perpetual struggle for money and talent, endpoints that foster reductive metrics, such as high-impact publication. National assessment procedures, such as the Research Excellence Framework, incentivise bad practices. And individual scientists, including their most senior leaders, do little to alter a research culture that occasionally veers close to misconduct.

https://www.thelancet.com/journals/lancet/article/PIIS0140-6...

How do you see this as scapegoating? The headline specifically says "intensifies", the article very clearly positions AI-fabricated data as an extension of existing problems, and I don't see anything in the article downplaying those existing problems (the entire closing section is about how the summit was on issues broader than AI).

Nature's reporting on the problem of paper mills is surprisingly high quality and honest, given that these reports directly attack the credibility of Nature itself (and many other journals).

Does the argument attack Nature and other journals or does it setup a position that makes journals even more important to provide filters, verification, etc. services?

What were facing is an explosion of Brandolini's law on a scale people are not prepared to deal with and when all financial incentives promote this direction and financial incentives rule the world, I'm not sure how we get around the issue in environments that support free speech.

I'm not proposing we curtail free speech but we have serious issues to deal with as a society in terms of the believability and sheer volume of false information. To some degree this has always been an issue but my concern is that there's a critical tipping point in a free speech environment where there's so much BS out there people completely stop believing any information not matter how reputable or valid it is.

It could make journals even more important, but in practice the reason paper mills exist is because journals aren't doing much (visible?) QA on papers, so spamming them with fake claims and auto-generated papers is a viable business model.

Journals unfortunately aren't stepping up to meet the challenge. I was writing articles about this problem several years ago and things haven't noticeably improved. This thread is full of people pointing out the obvious solution - take claim audit seriously and start by paying people to do professional peer reviews. Their actual solutions tend to look like spam filters for paper submission queues. It's enough to be able to say they're doing something, but not enough to actually make a big difference especially post ChatGPT.

I'm not so pessimistic, I think people learn to discriminate between sources. A lot of people right now are learning to generalize to "experts aren't" but it's a rather more nuanced understanding than the media like to make out when you dig in, for instance people understand that "expert" in this context usually means public sector funded academic or civil servant, and not e.g. a roughneck on an oil rig, some UI programmer at Apple, the guy who fixed their car last weekend. They learn that the first form of expertise is the type where making false claims can be beneficial for the people making them, whereas if you lie a lot about oil on an oil rig eventually something will explode.

I think we'll eventually get to a place where incentives are better aligned, for instance where data collection and aggregation is fully divorced from the people analyzing it. A lot of the distortion in science comes from the fact that academics need to collect data but aren't rewarded much for doing it, so once a dataset is collected it gets kept secret and that allows for a lot of dishonest game playing. It also means people are strongly incentivized to see things in the data that aren't really there. If data collection and analysis was fully divorced, those issues would go away (you'd get other problems of course but would they be worse?).

In science it seems the adage "talk shit get hit" is coming home to roost.

I would note there is a difference between untrue and fraudulent.

"Afflicted by studies with small sample sizes, tiny effects..."

Neither these are inherently the result of malfeasance or fraud, or even poor statistical practice.

Sample size is a compromise between a number of things - statistical power, yes, but also trying to minimize the number of human or animal subjects involved, and to be frank, budget (as I've noted elsewhere, the NIH R01 non-modular budget hasn't changed since the 90's). Before there's a body of work done, statistical power is often speculative. What do we think the effect estimate will be. If we're lucky, maybe we have a mathematical model suggesting at least something. In that arena, it's likely that we may undershoot the needed sample size - though I'll note underpowered findings are null findings, and far less likely to get published (and likely hopeless in say...The Lancet). There are also some questions that we may still want answers to where the sample size is inherently small. There are a finite number of Ebola outbreaks, or veterinary clinics in the U.S. (both real examples).

Similarly, tiny effects are hard to estimate, but that doesn't mean they're bad to estimate. Something that increased the risk of death in American citizens by 1% for example, would have a relative risk of 1.01, which is as small as many medical and epidemiology journals are apt to report. Yet this would impact thousands of people. Measuring that may be very hard, and very noisy, producing a number of wrong answers, but it's not self-evidently a bad idea. Especially if we don't know if the effect is tiny ahead of time.

> "The idea that peer review is enough to ensure that research is valid is perhaps not scaling well."

As others have said, this is not what peer review has ever been for, at all. It only checks for gross omissions and violations of form and syntax that are obvious to other scientists who are in adjacent fields (not even necessarily the same one). It's a relict from the times when publications were not target metrics. Peer reviewers never replicate the work. If you are a grad student tapped for peer review and you spend the time to replicate the work, you are ruining your career and your advisor will also be mad.

I have had reviewers and editors dig throughly into work, including code, and it has never been met with anything other than appreciation. I've sent reviews in detailing what further test or figure might be needed to make a point more persuasive, or noting inconsistencies, and I encourage my students to do the same.

> I have had reviewers and editors dig throughly into work, including code, and it has never been met with anything other than appreciation.

Yes obviously that kind of free and unexpectedly diligent labor would be appreciated.

> I've sent reviews in detailing what further test or figure might be needed to make a point more persuasive, or noting inconsistencies, and I encourage my students to do the same.

Yes these are the kinds of things peer reviewers do. Notably not replicating the work.

"Yes obviously that kind of free and unexpectedly diligent labor would be appreciated."

Then perhaps characterizing doing work - including often doing partial replication, independently deriving results, checking code to make sure it's both complete and consistent, etc. as "ruining your career and your advisor will also be mad" is unfair.

"I've sent reviews in detailing what further test or figure might be needed to make a point more persuasive, or noting inconsistencies, and I encourage my students to do the same."

Some of this is replication. "I cannot get from X to Y in your paper without the addition of Z" is, at its essence, a statement that a result cannot be replicated given what has been provided.

> "Then perhaps characterizing doing work - including often doing partial replication, independently deriving results, checking code to make sure it's both complete and consistent, etc. as "ruining your career and your advisor will also be made" is unfair."

You are mischaracterizing my argument in multiple ways. What I said was "Peer reviewers never replicate the work. If you are a grad student tapped for peer review and you spend the time to replicate the work, you are ruining your career and your advisor will also be mad." which I stand by. I don't think it's unfair. Maybe your reinterpretation is unfair, but I can't speak to that because you were the one who wrote it not me.

Then in response to you saying "I have had reviewers and editors dig throughly into work, including code, and it has never been met with anything other than appreciation." I said "Yes obviously that kind of free and unexpectedly diligent labor would be appreciated." which I also stand by.

In response to you saying "I've sent reviews in detailing what further test or figure might be needed to make a point more persuasive, or noting inconsistencies, and I encourage my students to do the same." I said "Yes these are the kinds of things peer reviewers do. Notably not replicating the work." which I also stand by. You are saying that "Some of this is replication." Well it's obviously not. Asking for further tests and figures and noticing inconsistencies are exactly the kind of things that peer reviewers usually do, and it's not replicating the work.

(PhD student in STEM here) I think most people have the wrong idea about what peer review is. My advisor teaches us to treat it as a first check, but it is not a guarantee of correct results.

Most of the time I spend on research is actually trying to understand the literature and reproducing their results. If I can't do it, it probably means I don't understand enough about the work I'm reading, but there is also the small chance that the published analysis is wrong, which already happened to me.

EDIT: typo.

Most people have the wrond idea about what publication is, too. For almost all papers, for almost all readers, it doesn't matter if results are correct. It's just part of the game of academia. If you need to build something based on a paper, and you need it to work correctly (so, outside of public policy or macroeconomics), you need to do the work yourself.

But if you just want to cite the work and write your own paper on top of it, go ahead.

Published science is like ChatGPT ;-)

scientific publication is like a JIRA ticket.

to get visibility of your work you need to close JIRA tickets, the more the better, since people look at aggregate metrics.

The more your JIRA tickets are quoted in company documentation the better.

but JIRA ticket is not code, in the same way publication is not science

It's never going to be a guarantee, but it's also implemented very shittily at present. There's a difference between what peer review is and what it should be. It makes sense to look at it realistically in day to day practice, but in discussion of systemic problems it should be viewed in a different light.

Peer review cannot catch if they performed the procedure correctly (e.g. in a lab), but it is there to check things like:

- Validity of the experimental setup

- Validity of the statistical analysis methodology

- Validity of the conclusions

Take a look at the items in this comment:[1]

> Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses

All these can be caught with peer review.

> together with an obsession for pursuing fashionable trends of dubious importance

In contrast, this problem is intensified with peer review.

> In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world. Or they retrofit hypotheses to fit their data.

Not the role of peer review to catch these, but making your data and analyses scripts open will allow these to be caught by anyone who wishes to. The problem we have with current publishing is that I could use this technique to get faulty results, publish in a prestigious journal, get a 100 citations on my paper, and one of those citations will be from the person who looked at my data and saw obvious biases in my selection (which cannot be caught by a mere peer review). That one citation pointing out how wrong I am is lost in the noise.

We need a way to highlight that citation - merely making a simple graph of connections via references doesn't get us there.

[1] https://news.ycombinator.com/item?id=36140540

While you should never treat anything in the literature as having "a guarantee of correct results", it should be a careful first check.

It might help to expand on "bogus". Bogus has a few levels, going from "not good, but possible from a well-intentioned author trying to do the right thing" (low-level bogus) to "outright deception" (high-level bogus). Small sample sizes, statistical errors, and flawed (but honest) experiment design are all, I suggest, low-level bogus. Faking data and plagiarism are high-level bogus.

I think peer review is capable of, eventually, mitigating low-level bogus. The quantitative standards in fields where low-level bogus is a problem (e.g., but definitely not only, medicine) are rising. Peer review is not a scalable solution to high-level bogus. Figuring out high-level bogus seems to be almost a full-time job [1]. You cannot expect this level of effort from researchers, especially if they are reviewing for free; I would even argue that it's easier to fake data than to figure out it's fake. It also requires more expertise to assess quality research than to write a low-level bogus paper and submit it. There's a mismatch here. There are not enough expert reviewers to handle all the low-level bogus papers.

The solution therefore seems to require some kind of reputational component. There needs to be a cost to engaging in high-level bogus. But this is a hard problem. Do you ban any lead author of a paper with demonstrated high-level bogus? Publicize their names? Ban any author of a paper with demonstrated high-level bogus? Throttle the submissions any one person can make to a conference/journal at a time? I don't know. But the current model will have to change.

[1] https://www.ft.com/content/32440f74-7804-4637-a662-6cdc8f3fb...

It's also possible to be functionally bogus while doing everything 100% by the book. If you control things super well you can inadvertently make your result so narrow that it's basically meaningless, at least with the way the scientific system functions today. If people worked together in a more concrete way to build on prior results this effect might not be so bad.

One of my favorite examples is a mouse study that did not replicate between two genetically identical mouse populations raised in the same conditions run by the same lab. The difference was the supplier the two sets of mice originally came from, and the researchers were able to pin down the cause as differences in the gut microbiome between the two (and in fact one particular bacterial strain). That is an example of great research, but the vast majority of studies will never catch something like this before they publish because they will only use one mouse supplier as they keep things controlled while minimizing costs.

Because designed replication studies are fairly rare and people often do not officially publish when they find things that don't replicate in biology, we are approaching interpretation/downstream use of these highly controlled studies in an extremely inefficient way.

But that isn't really the fault of individual researchers. Technically they're applying the scientific method correctly to their niche problem. It's the definition of the problem coupled with how we combine results across groups that causes the inefficiencies. As problems of interest increase in complexity we can't define them on the scale of individual labs anymore, and for some reason we've addressed it by breaking them up into subproblems in this homogeneous way. Then we just assume that piecing together the little results will work great...

Anyways, I agree there is also straight up fraud and blatantly bad practices on the level of individual papers, it's definitely a continuum. Sometimes such bad results slip into the mainstream of science or even have a huge impact on subsequent funding directions like with the fabricated amyloid beta paper. But I do suspect that for the most part the blatantly bad work stays on the fringes, and the largest negative impact on scientific productivity actually comes from a level of abstraction up.

The way academic research is setup is not currently conducive to effective research. Research must intrinsically be allowed to fail. If you incentivize success, then what happens when someone is 2-4 years in and their only skill is apparently worthless?

Without Tenure or the ability for a scientist to accumulate their own savings, there is a strong incentive for most scientists to charitably interpret results/papers to maintain relevance.

Imagine a world where all the raw data sits in a public repository next to a script to process the data which was reviewed as part of the publication process, which would accompany the paper and could be quickly and easily replicated with new data. Anyone could produce the graphs shown in the paper simply by downloading both and running them together. What a wonderful thing that would be. And you could imagine the government funding the storing and serving of all the data. What a wonderful daydream of an idea.

seems attainable, doesn't sound far fetched technically. Maybe get a few universities to start the trend.

Even this would require somehow verifying the raw data. It's plausible a bad actor could "reverse engineer" their data from a pre-determined conclusion.

But yes, overall more openness is good. Still, the cost losing trust in society is very high (as you need to verify everything).

> It's plausible a bad actor could "reverse engineer" their data from a pre-determined conclusion.

I've already heard of someone planning a product ( initially targeted at lazy^H^H^H^Hbusy high schoolers and undergrads) that will use AI to reverse discover citations that fit a predetermined narrative in a research paper. Write whatever the hell you want, and the AI will do its best to backsolve a pile of citations that support your unsourced claims and arguments. The founder, and I use that term very generously, expects the academic community to net support this because it will boost citation counts for the vast majority of low citation, low visibility works.

Some journals in fact implement this idea (e.g. having the raw data underlying each figure one click away). That is however not the crux of the reproducibility crisis; it would be great if it was just "I can't make an exact copy of Supplementary Figure 5D", but rather "I can't confirm that protein X causes dementia using orthogonal techniques". There is no easy code fix for that problem.

Funding the storing and serving of all of that data doesn't sound like a difficult problem to me. That has gotten SO cheap over the past couple of decades. There are plenty of well funded institutions that can support that kind of resource.

You'd be surprised. One headache there is what did you tell study participants you would do with the data?

Did you say you'd keep it forever? Did you say 5 years? Who's in charge of making sure that this centralized repository isn't inappropriately holding and distributing data?

Funding organizations also have different requirements.

Then a script is only useful if paired with a set of libraries of a particular version, a specific compiler/interpreter, an OS, also there may be specialized hardware involved. Some of the languages used in science like SAS, Stata, SPSS, and Matlab, etc aren't free and open source so you can't always just bundle it.

And even data storage isn't trivial. For a recent small conference abstract I processed ~150GB of data. Hundreds (thousands? Tens of thousands?) of other papers have looked at that same data. You would really want some way to deduplicate that storage, but that introduces some additional complexity.

I do like this vision but I think it would be a major undertaking that would require a lot of well funded institutions coming together rather than any one in particular doing it on their own.

This.

"Your data is open and available in perpetuity for whatever use" is in deep conflict with how we think about human subjects data, often for very good reason.

One thing that was really jarring for me moving from the startup/consulting/advertising space to the research space was that human subjects data really gets deleted.

It's not just the deleted_at column being set, there's no backup, it's really gone. Every copy, forever.

I appreciate the ethics of it, and part of my reason for working in this area is because of these ethics, but even 5 years in there is so much reluctance to press that delete button.

This is an excellent comment, thank you.

I can only really speak to astronomy, but the biggest problem is generally not hosting the raw data. The problem is maintaining the process of analyzing the data.

When you are doing science, you often do not just have a single standardized data format. It's not like taking a picture with a camera, where the jpg format is a standard and the metadata is a standard. If you are, say, storing data from a radio telescope, the metadata is more like taking a snapshots of a production database. Over time you might track additional data, like how much the telescope slewed recently, how much radio interference was nearby according to some new algorithm, etc etc.

Your data formats are constantly changing. So your analysis scripts are constantly changing as well. But this sort of thing doesn't just maintain itself. You very often need to add code like, how do you handle versions of the data where column X is missing. A research project might spend a year gathering data, and change the data format a tiny bit ten times over that year. If you publish something a few times a year, there eventually are a huge number of data versions and script versions that old publications rely on.

It isn't impossible to maintain data like this. You can have code that regularly runs integration tests and reruns past analyses. But most research doesn't operate to this level of "software engineering quality". One-off Jupyter notebooks, code that the developer only got working on their local machine, and so on.

I think we could do better, but it would involve hiring more software engineers and building engineering teams to support scientific research. It is not as simple as allocating more budget towards hosting large files.

I hate to be sincere, but the reality is the data is our product. If we open source our data too much, we won't have anything left to publish as others who would have to expend nowhere near the resources we must do to produce it can scrape it and publish it. (that literally is how "AI" of the current hype bubble works today, lol, why would I want that online?)

The incentives are definitely bad, and that's where the actual fix should be.

If any level of government funds your institution you should have to releases the data.

The code I write is not my code, it's the banks.

If that happened it would indirectly change the incentives anyway, because everyone would start being required to release data, so bad practices that are currently incentivized would become impossible.

However I think it would be better to directly incentivize data release rather than require it, at least in biomedical sciences. Because of patient privacy issues there is no way raw data release can be required across the board. And I certainly do not trust the NIH to come up with a coherent set of rules for when it is versus isn't allowed, which would mean loopholes and more corruption.

Most projects are funded by a patchwork of different sources, not all of which would agree to the same terms of release of the information, but all of which are required sufficiently fund its creation. Not to mention maintaining ongoing storage and accessibility. single source government grants for scientific research are the exception, not the rule.

What if I use your medical records, which contain pretty trivial identifying information in them?

I don't think this is the main reason people don't open source data. If your data is high quality and others are using it to find interesting stuff then you are going to get citations and authorship on additional publications with minimal work, which sounds like a great ROI. The reason data doesn't get open sourced is because researchers are not actually confident in the data and don't want their other works to be exposed.

"If your data is high quality and others are using it to find interesting stuff then you are going to get citations and authorship on additional publications with minimal work..."

This has not been born out in my experience, or the experience of others. Data products are chronically undervalued and undercited in science, and do not come with guarantees of authorship unless you put up barriers to access them without it.

Open data is, at this point, a decision made for either ideological reasons or as a condition of funding and/or publication, not a decision that is itself usually "worth" the investment of curating a public data set.

This is the limit to what people do. In my field, even source code sometimes isn't free software. However, if you ask a researcher, they will give you a copy you can freely modify, because the act of asking means they know who you are as a person, and can later ask for citations when you publish. Data is similarly treated. Just putting it out there without any barriers guarantees that other parties will use it without citing you.

The unfortunate truth is that academic lineage matters. In a world with decreasing SNR from paper content, the SNR from finding the paper on a reliable researcher's homepage massively increases (because sadly, people are added to papers without their permission in an effort to boost their own signal).

> Most scientists cannot even reproduce their own results. It's common to use things like manual Jupyter notebooks, unlabeled CSVs, and a bunch of disorganized data files, in a one-off process that a scientist manually summarizes to produce a paper.

The only way to get that under control is if universities had a career track of data scientists and programmers, basically a shared resource pool of specialists that all researchers could use. But most US universities seem to prefer to "invest" their endowments into professional athlete teams.

No offense, but astronomy is ripe for being rife with this sort of thing because the validity of astronomical research doesn't matter for the "real world" and can't really easily be tested. I feel like it could be better, but since you guys do not have any external pressures beyond yourselves (as academics), you can still publish however you feel like.

A lot of the more applied fields cannot survive scrutiny. Either you have to produce something that leads to a product or something a PM will scrutinize (especially if you work for a national lab or the like, or for pharma, and so on). There, a lot of published research might be bogus, but it is nowhere near the majority.

Astronomy is fantastic for preventing this type of fraud.

The data is incredibly open. Anyone can download datasets from telescopes, or build their own. Papers built on invalid data can be refuted by simply pointing telescopes at the location the paper is lying about. There are very few possible discoveries or theories in an astronomy journal that would immediately impact the wider culture.

Compare astronomy to sociology, in which questionnaire results can be fabricated and are essentially unconformable without using other unconformable questionnaires. Compare astronomy to economics, which the articles can be used bolster political positions.

My instinct about numbers (i.e. incl. statistics) is that they should be reproducible: on-demand, with results and workings easily subjected to interrogation.

In reality in many fields - my experience is in finance, but also in science - calculations are scattered across different languages and systems. This adds friction to any process about reproducing, understanding, analysing numbers.

This is some motivation for calculang, a language for calculations I develop. https://github.com/calculang/calculang

For the HN crew it's an under-development(!) functional language with properties to permit flexible designs that can scale. It's for numbers and if you share the model alongside your numbers people can check them according to the model and see the workings.

It goes well with visual number [dev/person]tools that I will release one of soon, and in the future watch for a browser extension for the workings behind numbers you are reading.

Important, it won't address the raw data part of the problem, but where numbers following from that are concerned, it might get closer to that instinct.

I love the idea. But do be aware that the high-quality software engineering like this will require has a cost: be prepared for basic science costs to scale appropriately (or for productivity to be reduced.) More to the point, not all science can be usefully verified by unit tests. Even machine-verified mathematical proofs are entirely dependent on the definitions being correct, and that requires expert human analysis.

Given the non-modular budget for an NIH R01 has not been updated since Clinton was president, there is absolutely no preparation for basic science costs to scale appropriately with inflation, let alone adding something that is a lot of work for relatively minimal per-paper return.

It's ironic and deeply shameful when whitepapers side-step the most fundamental aspect of the scientific method: reproducibility.

I think if the data is valid, open access, and reproducible, it shouldn't matter if the paper was written by AI.

The article says this:

> Kahn says that, although there will undoubtedly be positive uses of AI to support researchers writing papers, it will still be necessary to distinguish between legitimate papers written with AI and those that have been completely fabricated.

Unfortunately this assumes that the only thing which can go wrong is lack of reproducibility. Not so. I read a lot of public health papers during COVID and a staggering quantity (IMHO nearly all of them) should not have been published; many of them would have been reproducible despite that.

Other things that can and do regularly appear in reproducible, peer reviewed papers:

• Nonsensical methodologies

• Logical fallacies

• Mis-representation of their data

• Incorrectly implemented software

• Source datasets that are cherry-picked

One might think that things like incorrectly implemented software would fall under the umbrella of irreproducible research, but that won't work. Some fields don't really recognize a distinction between model implementation and specification. The model is the implementation and if a description was once published it's quite possibly either too vague to implement, out of date, wrong, or all three. IIRC Prof Neil Ferguson's team actually rejected an attempted replication of one of their epidemic models on the basis that only they were qualified to use it! Pseudo-science like this goes unremarked in universities, only outsiders seem to care.

Sometimes you get an honest academic who knows what they're supposed to do and actually does it, but there's no observable benefit to them from doing so because the ones who don't bother don't seem to suffer any consequences.

tl;dr The biggest problem with the replication crisis is the framing of it as being about replication. What the world actually faces is a misleading research crisis. You can drive the replicability rate to 100% and you'll still find whole fields consisting of logical fallacies and misinformation.

I think you'll be demonstrated to be wrong in short order over a period of the next 1-2 years.

My called shot: We'll have fully automated scientific research with useful and valuable conclusions and little to no human intervention. I think it will take some time, but can tell you right now its on its way to happening.

Reply on news.ycombinator.com