Hacker News Reader

Comment by renonce | Hacker News Reader

I think needing LLM summarizers to read a paper at all only highlights the failure of paper authors writing the abstracts. Let's face it: Abstracts are getting intentionally more complex and hard-to-read or reviewers will question the paper's writing. If the LLM summarizers were useful, the authors could have generated it and just used it in the paper's abstract section. And indeed, there is no better person than the authors to edit the LLM generated summary because they know what parts of the summary is hallucinated and what is not, right?

View on HN

An abstract generally has the following format, it starts by describing the background of the problem, the problem the paper aims to solve, the method the paper uses, and finally a conclusion. The abstract doesn't assume much prior knowledge, and can probably still be understood 10 or 20 years from now. Whereas you can see how the LLM summarized version totally skips the background and jumps straight to the problem and the method.

Now, I'm not saying there is no room for improvements. The fixed format an academic paper has with abstract and the actual paper may actually be replaced by what is shown here, and I genuinely hope to see more experimentation with the communication of scientific studies, but that is unfortunately not being focused on in the academic world.

It makes sense to debate what should be included in the abstract. Should background, problem, method or conclusion be included? My personal preference is to read the problem and method only because that’s what gives me inspiration and helps me decide whether the paper is relevant. I acknowledge everyone may have their own preference, and as mentioned in other comments, a major feature of LLMs is that you can fine-tune it using instructions to decide the level of detail that you want. But I think the main contention is that the paper authors could have done just slightly more work beyond getting the paper accepted to have the paper reach a much wider audience than their specific field.

I think it matters who we want to write papers to. Right now we write papers and abstracts to reviewers. That's because that's how we're measured and that's where we compete. But I'd say that we generally believe that papers are written to other researchers, which I agree that that should be the goal. But as this competition is increasing we're starting to write more to media as this can usually pass review and gathers lots of citations (these people tend to be from big schools too which have large media arms and are willing to pay for articles in news venues).

This is why I'm deeply frustrated with academia right now. Papers are supposed to be how I communicate to my fellow researchers working on the same or similar topic. They're not for communicating to someone in a different field and not for communicating to the public layman (nor should they be!). It is the job of science communicators to act as the bridge between laymen and researcher, which a lot do a poor job as they're beholden to the YouTube algorithm, not accuracy. Hell, Quanta published a shit piece recently about quantum wormholes and machine learning and what did they do when it was called out? Just write another article and add a note on their youtube video. Nature is pulling similar shit. I get wanting to make science popular and exciting, but truth/accuracy has a lower bound in complexity whereas fantasy doesn't.

https://www.quantamagazine.org/physicists-create-a-wormhole-...

https://www.quantamagazine.org/wormhole-experiment-called-in...

https://www.youtube.com/watch?v=uOJCS1W1uzg

I really wish they would limit abstracts to those. I feel like abstracts should be like an index/jumping off point in spirit if not in structure

There's a larger problem with writing papers, that results in this problem that I discuss here[0] and it looks like you're getting at too. What I'd say is that authors (being one myself) are writing papers to reviewers, not to other researchers in my niche (sometimes it feels like not even my several levels of abstraction above).

I can confirm having to significantly tweak papers in ways that I would not have done writing to other researchers. I have papers with hundreds of citations as well as top benchmark scores on papers that could not pass reviews with the most common complaints of "not novel" and "I don't know who would find this useful." This has been one of the most challenging aspects of my PhD and certainly one of the most frustrating.

But the larger problem I see is that everyone is simply hyper-hacking every metric that they can. This is beyond academics, I'm sure you see it in your work or politics too. I think we need to have a serious discussion as to the fact that metrics are proxies and not always aligned with our goals. Or that they stay aligned with our goals, because if someone gets an advantage by optimizing towards the metric rather than optimizing towards the abstract goal we actually want.

It's not AI turning the world into a paperclip that we should be afraid of, it's humans doing that.

[0] https://news.ycombinator.com/item?id=38200598

In math, the job of writing a comprehensive summary is usually done in other places. Some of them are even accessible for free, such as zbMATH [1], a big database of comprehensive summaries of math articles.

[1]: https://zbmath.org/about

All of it is hallucinated, some times it happens to be accurate, some other times it is not.

Reply on news.ycombinator.com