Comment by raddan | Hacker News Reader

Comment by raddan | Hacker News Reader

Parent Full thread

> That is not realistic, but I suppose where things are heading ...

And to expand on this, it's not realistic because science is not armchair philosophy. You have to go out and measure the world.

Sometimes, through force of will, a person can think deeply about a problem and come up with beautiful theories that explain our measurements. Many scientists had careers like this, probably most famously, Einstein. But it's worth noting that Einstein also got a lot wrong! [1]

Even if we somehow give an LLM the ability to go out and measure things, I seriously doubt that the role of humans in science is done. There's a big difference between "an explanation" and "a good explanation." Ask any physicist. There's a surprising amount of aesthetics involved. Good theories are consistent with the evidence, but it's more than that-- there's a great deal of "taste" involved. And there's a good reason for that. For any real problem, there are effectively an infinite number of alternative hypotheses. From a "theory of science" standpoint, this should cause scientists nightmares, but it doesn't. Because by the time you are a practicing scientist, you've developed a feel for what constitutes a satisfying explanation. If you spend time with scientists, especially in the "hallway track" at a conference, "taste" is a frequent topic of conversation!

[1] https://en.wikipedia.org/wiki/Einstein%27s_unsuccessful_inve...

I think you misunderstood. I'm not asking for an oracle that can determine whether a paper is correct, I want an oracle that can find real mistakes in papers (thus invalidating them).

I work full time on "lab in the loop" AI, so I'm pretty familiar with the need for real-world experiments. I am not proposing a fully autonomous scientist that could read an arbitrary paper and emit whether it's universally true without some verification method.

Also, to your statement: " Because by the time you are a practicing scientist, you've developed a feel for what constitutes a satisfying explanation."

I'm a practicing scientist (well, ex-scientist) and it seems like most "satisfying explanations" end up being wrong or incomplete simply because they seem so satisfying.

> I'm not asking for an oracle that can determine whether a paper is correct, I want an oracle that can find real mistakes in papers (thus invalidating them).

What's the difference? How do you find real mistakes without a model? Either you have a trusted mathematical model (in which case you already have a complete explanation) or you have to compare it against the ultimate oracle: the world. Or are you proposing something like "let's use an LLM to convert this hand-wavy English paper into a formal proof and then check it for logical fallacies?" In which case, fine, that would be useful, but that's not exactly the same thing (and also not as important) as saying that a paper advances a bad explanation. Just that the explanation is flawed in some way.

The most common and obvious example is image duplication. This is used to invalidate large numbers of paper (I was absolutely shocked at the observed rate of image duplication). I am not sure I would call that a model.

The next example I can think of- I am not sure it qualifies. I read a paper where they deleted one gene at a time in yeast (it has 6000 genes) and determined whether the mutated yeast could live or not. For each gene where the yeast died, they added that to a list of "essential for life" genes. The paper concluded they had found some interesting proteins that should be studied. I read the paper and the first thing that sprang to mind, are any of these genes overlapping? Because we know (somebody already demonstrated in a lab) that genes do overlap (which is truly weird!)

I wrote a script and showed that every gene they reported as essential for life overlapped an already known gene that was essential for life. I wrote the authors, who never responded, but wrote a followup paper where they acknowledged they probably had a high false positive rate due to overlapping genes with known fatal effects. My guess is you'd say that either I used a model (existing literature) or I compared against the world, but realistically, what I did was trivially come up with a better explanation than the authors. T hat's what I want LLMs to do for me, and it seems like the direction LLMs are going will fulfill my desires.

How do you determine that a mistake is serious enough that it actually invalidates the paper? How do you know that it's not possible to correct it and reach a similar conclusion with fundamentally the same approach? Especially when the fix would be complex enough to justify a follow-up paper.

You can get a good idea by reading Retraction Watch and following a few papers/scientists who show up in it. That defines the current norms.

To me, a single significant error of any kind brings the entire paper into question. If a less important figure contains an image duplication, that makes me wonder if I can trust any of the images.

That's the administrative aspect of the question. But as a scientist, I'm more interested in knowing if the result is real. Sometimes even fatally flawed papers make valuable contributions, because the proposed approach will lead to the claimed result, once done properly.

I'm not aware of any fatally flawed paper that leads to the claimed result when done properly. Could you share an example?

It depends on what you see as a fatal flaw, and if you count mathematics and theoretical fields or just actual science. In mathematics and theoretical computer science, it's pretty common that major results are initially wrong, and fixing the errors can take a long time. Wiles's proof of Fermat's Last Theorem is the best-known example.

In any case, my point was that rejecting results due to technical flaws is intellectually lazy. As a scientist, your job is more about trying to find value in other people's work than finding excuses to reject it.

I can't speak to math and theoretical CS, because both of those fields now have methods to make proofs that can be verified (and math is probably the only situation where you can "prove" something true; all other fields are effectively probabilistic, not logical in nature).

I disagree with your premise. Part of our job as scientists (thankfully no longer mine) is to reduce the irrelevant and incorrect noisy as early as possible. I have seen so many grad students get excited by a paper and put enormous effort into reproducing somethign that was a false or fake result.

As scientists, we need to have effective filters to quickly ignore things that are probably not relevant to us. But we also need to acknowledge that those filters are necessarily noisy, and they don't tell much of the value of the work they reject.

If you want to determine reliably whether something is irrelevant and incorrect noise, determining whether there is anything of value is a necessary first step.

I've seen many reproduction attempts in bioinformatics fail, because they person trying to reproduce the work didn't have the conceptual background to do it correctly. Instead of spending enough time studying the theory, they rushed directly to action.

Reply on news.ycombinator.com