Hacker News Reader

Comment by tobias2014 | Hacker News Reader

tobias2014

Why would someone ruin their academic credentials and reputation (at a reputable university) like this? Maybe there was nothing to begin with? Otherwise I can't understand this.

View on HN

I think it might be taking a position in the use of AI, They don't appear to be concealing their use.

If the papers were of a uniformly high standard then I might not have a problem with it.

If they are hoping some of them are good and the review process will filter out the bad then it seems like they are not doing research, they are writing hypotheses and at pushing the work of testing them onto the reviewers.

I could also see that being done to make some sort of statement about the review process, but it's not clear if that is their goal.

He's wasting people's time, which is already the resource that many people lack any to spare. I'm not sure if he realises this part.

Could probably write a real research paper based on the acceptance outcomes of all of these LLM written papers.

This is economics though, a discipline where you be like Kenneth Rogoff and Carmen Reinhart and commit the most impactful fraud of the century and face absolutely no consequence on your career.

When you compare that to the consequence of the hydroxycholoquine paper, which is comparable in public policy impact, where Didiee Raoult lost all its legitimacy and his job, it's easy to see that economics as an academic field doesn't value probity much.

It's incentivized by the system. You get grants, clout, importance, job opportunities based on your publishing in major conferences. As of last year, the variance of review scores have been higher between reviewers on the same paper than between the same reviewer on different paper. So it's effectively become a coinflip. The best strategy when it's random is just to spam stuff.

Oh also at least 40% generated peer-reviews are contributing to this statistic.

Because they won't. The various algorithms that create the "publish or perish" incentives actually reward this. Academic reputation isn't a term in those algorithms, so it isn't rewarded. Same for verifying other's papers and other parts of science and research that aren't actually publishing papers. This is the cause of the reproducibility crisis in academic.

"Show me the incentives and I will show you the outcome"

Reputation is a factor, because most processes involve a subjective evaluation by other experts in the field. And non-experts use service activities such as peer review as a proxy for reputation. If reputable entities use your services as an expert, you are probably a reputable expert in the field.

Not surprising. Many academics held in high-esteem in these behavioral offshoots are nothing more than frauds to begin with. Almost every X-studies paper is equally fluff these days (see the recent frontpage CHI post) - at least these have some semblance of theory given their equations, though are still largely garbage.

More and more evidence is surfacing that suggests AI addiction is real. It may be as simple as that Nicholas Polson has developed a severe case of AI addiction.

Maybe his university is mandating or threatening to fire him like half of America’s corporate tech companies are.

If amoral, they could have been emboldened by exposure to AI use, either their own or by other students, colleagues, etc., while lacking inherent moral pressures for/against slop.

If moral, they could have decided that propagating their message will improve the world, even if only at x% efficiency due to use of AI, and that their intentions will take precedence over being accused of slopping.

If immoral, they could have chosen to run a grift on the chance that no one would notice, overdone it, and then played every card in their hand while perhaps sweeping away evidence of their slop once caught.

(These are constructed teaching examples, not situation-specific inferences or claims.)

A ton of academia is based on putting out slop into not so well-read journals and crossing fingers it passes some dead-exhausted postdoc TA's peer review.

AI psychosis? They think they're actually putting out high-quality papers. Nothing else makes sense.

Maybe to expose the utter uselessness of "papers".

I'm not an academic, and I understand by osmosis that organized science has many problems related to publishing and connected mechanisms, but I have to say there's been quite a few moments in my life and career when I found the inspiration or solution I needed in a good paper.

For condensed knowledge on highly specific topics (i.e. more specific than is economical in book form) I often know nothing better.

Most recently I did sort of a self-taught crash course in ground-penetrating radar and applications for an archeological endeavour I write software for, and a number of papers have been really invaluable.

I think one of the end-game components for AI is to be able to read all the literature and generate a reliable list of which ones are not correct and a convincing reason why.

> generate a reliable list of which ones are not correct and a convincing reason why

That is not realistic, but I suppose where things are heading is that you have some indicator of the strength of evidence -- see Fig. 13 in the following insightful take:

https://news.ycombinator.com/item?id=49407226

Though, "strength" should probably be "reliability" and "validity", and I suppose those indicators are more for picking signals from the noise; i.e., what is even worth clicking and reading. That would be increasingly valuable already today due to the volume (and, yes, slop and other related stuff).

> That is not realistic, but I suppose where things are heading ...

And to expand on this, it's not realistic because science is not armchair philosophy. You have to go out and measure the world.

Sometimes, through force of will, a person can think deeply about a problem and come up with beautiful theories that explain our measurements. Many scientists had careers like this, probably most famously, Einstein. But it's worth noting that Einstein also got a lot wrong! [1]

Even if we somehow give an LLM the ability to go out and measure things, I seriously doubt that the role of humans in science is done. There's a big difference between "an explanation" and "a good explanation." Ask any physicist. There's a surprising amount of aesthetics involved. Good theories are consistent with the evidence, but it's more than that-- there's a great deal of "taste" involved. And there's a good reason for that. For any real problem, there are effectively an infinite number of alternative hypotheses. From a "theory of science" standpoint, this should cause scientists nightmares, but it doesn't. Because by the time you are a practicing scientist, you've developed a feel for what constitutes a satisfying explanation. If you spend time with scientists, especially in the "hallway track" at a conference, "taste" is a frequent topic of conversation!

[1] https://en.wikipedia.org/wiki/Einstein%27s_unsuccessful_inve...

I think you misunderstood. I'm not asking for an oracle that can determine whether a paper is correct, I want an oracle that can find real mistakes in papers (thus invalidating them).

I work full time on "lab in the loop" AI, so I'm pretty familiar with the need for real-world experiments. I am not proposing a fully autonomous scientist that could read an arbitrary paper and emit whether it's universally true without some verification method.

Also, to your statement: " Because by the time you are a practicing scientist, you've developed a feel for what constitutes a satisfying explanation."

I'm a practicing scientist (well, ex-scientist) and it seems like most "satisfying explanations" end up being wrong or incomplete simply because they seem so satisfying.

> I'm not asking for an oracle that can determine whether a paper is correct, I want an oracle that can find real mistakes in papers (thus invalidating them).

What's the difference? How do you find real mistakes without a model? Either you have a trusted mathematical model (in which case you already have a complete explanation) or you have to compare it against the ultimate oracle: the world. Or are you proposing something like "let's use an LLM to convert this hand-wavy English paper into a formal proof and then check it for logical fallacies?" In which case, fine, that would be useful, but that's not exactly the same thing (and also not as important) as saying that a paper advances a bad explanation. Just that the explanation is flawed in some way.

The most common and obvious example is image duplication. This is used to invalidate large numbers of paper (I was absolutely shocked at the observed rate of image duplication). I am not sure I would call that a model.

The next example I can think of- I am not sure it qualifies. I read a paper where they deleted one gene at a time in yeast (it has 6000 genes) and determined whether the mutated yeast could live or not. For each gene where the yeast died, they added that to a list of "essential for life" genes. The paper concluded they had found some interesting proteins that should be studied. I read the paper and the first thing that sprang to mind, are any of these genes overlapping? Because we know (somebody already demonstrated in a lab) that genes do overlap (which is truly weird!)

I wrote a script and showed that every gene they reported as essential for life overlapped an already known gene that was essential for life. I wrote the authors, who never responded, but wrote a followup paper where they acknowledged they probably had a high false positive rate due to overlapping genes with known fatal effects. My guess is you'd say that either I used a model (existing literature) or I compared against the world, but realistically, what I did was trivially come up with a better explanation than the authors. T hat's what I want LLMs to do for me, and it seems like the direction LLMs are going will fulfill my desires.

How do you determine that a mistake is serious enough that it actually invalidates the paper? How do you know that it's not possible to correct it and reach a similar conclusion with fundamentally the same approach? Especially when the fix would be complex enough to justify a follow-up paper.

You can get a good idea by reading Retraction Watch and following a few papers/scientists who show up in it. That defines the current norms.

To me, a single significant error of any kind brings the entire paper into question. If a less important figure contains an image duplication, that makes me wonder if I can trust any of the images.

That's the administrative aspect of the question. But as a scientist, I'm more interested in knowing if the result is real. Sometimes even fatally flawed papers make valuable contributions, because the proposed approach will lead to the claimed result, once done properly.

I'm not aware of any fatally flawed paper that leads to the claimed result when done properly. Could you share an example?

It depends on what you see as a fatal flaw, and if you count mathematics and theoretical fields or just actual science. In mathematics and theoretical computer science, it's pretty common that major results are initially wrong, and fixing the errors can take a long time. Wiles's proof of Fermat's Last Theorem is the best-known example.

In any case, my point was that rejecting results due to technical flaws is intellectually lazy. As a scientist, your job is more about trying to find value in other people's work than finding excuses to reject it.

I can't speak to math and theoretical CS, because both of those fields now have methods to make proofs that can be verified (and math is probably the only situation where you can "prove" something true; all other fields are effectively probabilistic, not logical in nature).

I disagree with your premise. Part of our job as scientists (thankfully no longer mine) is to reduce the irrelevant and incorrect noisy as early as possible. I have seen so many grad students get excited by a paper and put enormous effort into reproducing somethign that was a false or fake result.

As scientists, we need to have effective filters to quickly ignore things that are probably not relevant to us. But we also need to acknowledge that those filters are necessarily noisy, and they don't tell much of the value of the work they reject.

If you want to determine reliably whether something is irrelevant and incorrect noise, determining whether there is anything of value is a necessary first step.

I've seen many reproduction attempts in bioinformatics fail, because they person trying to reproduce the work didn't have the conceptual background to do it correctly. Instead of spending enough time studying the theory, they rushed directly to action.

Haha, Sean Eddy used to complain about reproduction efforts in bioinformatics (specifically, people trying to benchmark HMMER and doing a bad job). Fortunately, my advisor helped me learn the techniques and Sean approved (he was also happy that HMMER beat BLAST for remote homolog detection).

I also left a postdoc position over my professor's decision to rewrite my paper to juice all the stats- not a specific error, but selectively interpreting the data to make the results look better than state of the art, when they were not.

I would absolutely love to have my "paper correctness AI" mark all those bad reproductions in the literature and outright misrepresentations- it's all too easy to rush to publish and get a lot of attention- especially if your advisor or coauthors are prestigious and mildly unethical.

This is realistic. We already do this today: it's called "journal club". A bunch of grad students read the same paper and then criticize it. I've read papers that I thought were amazing only to. have somebody else notice a key issue in a method, or a conclusion that didn't follow, or outright omission of an important detail, in a way that could be verified by both the students and the authors of the paper.

>... and a convincing reason why.

I believe this is one of the LLMs' more destructive elements. The more effective ones are quite good at providing output that gets the meat to repeat the output to other meat. Whether the output actually benefits the meat is disconnected from whether or not the meat accepts and repeats it.

Change the word LLM to "teacher" and I think the statement is still true.

You can be served this inspiration from LLMs

I've tried here and there, but even the top-end models always tend toward shallow generalist takes. I've not been able to get more than basic primers out of LLMs, and nothing close to the ability of a professional author to stay on course with an idea, detail level, specificity etc.

People in the early days used to often whine that LLMs just regurgitate text snippets (unfounded of course), but I think the way we currently train and RLHF them actually seems to largely make them unable to reproduce the knowledge they have been trained on, since they seem to just always want to please the mean with their output. I'm oversimplifying the mechanisms, but you get my drift.

But what prevents the workflow of:

You ask the LLM for papers in this special field that you are interested in.

It does the search, fetches 20 papers it finds relevant

Summarizes them or points to ideas with reference

Alan Sokal style hoax is what comes to mind first, of course. Wouldn't be the first computer-generated hoax either (see SCIgen). However see also the P.P.P.P.S. in the article, his co-author denies that in the comments.

If the past 30 years have shown us anything, it's that academic credentials and reputation are very difficult to blemish.

Polson is just pushing the knob to 11 because it's louder.

We've been in the dark ages of science for so many decades...

Reply on news.ycombinator.com