Hacker News Reader

Comment by OutOfHere | Hacker News Reader

To produce scientific work, one needs certain raw materials:

1. Data

2. Access to past works

Once you have these, only then can discoveries can be made, and papers be written. How does this software get these? I am assuming they have to be provided up-front to the software for each job.

View on HN

It will analyse what it is given, but it will not have the ability to say, "hang on, these results are interesting, I wonder what will happed if I pour a different liquid into the drum and spin it at the same speed?" LLMs, especially the latest ones, are decent at analysis of input, but disappointing at producing creative output. I am running a series of experiments using Gemini 1.5 and found it capable of producing good results if you stay away from "write me an academic paper on subject X" or "write me a novel". On the other hand, if you ask it to summarise text, extract particular information, it is fast and arguably good, but not necessarily great. It will miss things and miscategorise them requiring a human being to check its output. At this point, you may just as well do the job yourself. LLMs are still not very good, despite what their fans are saying. They are clever, as in a "clever trick" not as in a "clever human being".

Interestingly your comment is the very opposite of my experience with LLMs.

You can rely on them for creative stuff (write a poem, short story, etc.), but you cannot depend on them for factual stuff. Time and time again they will state "facts" that turn out to be false, and so now I no longer trust it for anything without manual verification. And since I then need to do the research myself for the verification, I rarely find LLMs helpful, except occasionally for initial exploration of some topic.

You used summarization as an example, but whether they are fundamentally good at that is even debatable, e.g. https://ea.rna.nl/2024/05/27/when-chatgpt-summarises-it-actu...

I revisit LLMs a couple of times a year to see if they have gotten any better and it's not a great experience, but I will admit they have gotten slightly better. Still lack and will lack ability to understand what they are processing.

Your experience contradicts the data from the paper. Sonnet generated almost entirely novel concepts, GPT achieved ~4/5 novel ideas. Maybe it's specific to the area of research and the way of prompting, but "it will not have the ability to say" seems to be proven wrong already.

To produce scientific work, one needs to follow the scientific method. This involves stating a hypothesis, designing an experiment that would test this hypothesis, conducting the experiment with controls, and analyzing the data w.r.t to the hypothesis being tested.

Access to past works is only useful in informing what is a good hypothesis worth testing. And data is only useful when generated by an experiment that is testing for causality (see [1]).

[1] https://pyimagesearch.com/2023/11/27/a-brief-introduction-to...

The kind of cargo cult science you describe is the main reason for the replication crisis. The more people believe that they will reach true knowledge by following the sacred rituals to the letter, the more likely they will do things in the established ways without thinking and repeat the same mistakes over and over again.

In actual science, the key step is stopping to question yourself all the time. Does the thing you were planning to do still make sense? Especially in the current context? Given what you have seen so far? Should you change your plans? If so, can the work you have already done be salvaged? Or do you have to discard it and start over from the beginning?

Excuse me, what?! How is this cargo cult science? I literally just described the scientific method. Here it is described on Wikipedia:

> The process in the scientific method involves making conjectures (hypothetical explanations), deriving predictions from the hypotheses as logical consequences, and then carrying out experiments or empirical observations based on those predictions.

The other questions you added are not bad questions to ask if you are practical and resource-constrained, but this is besides the point.

https://en.wikipedia.org/wiki/Scientific_method

It's cargo cult science precisely because you described the scientific method. Cargo cult science is about focusing on the form without understanding its purpose. Or in Feynman's words:

> So I call these things Cargo Cult Science, because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential, because the planes don’t land.

If you are using the scientific method because it's the scientific method, you are doing cargo cult science. The scientific method is a useful tool in a scientist's toolkit, but it's not appropriate to every situation. And even when it's appropriate, it's not a central part of doing science. You can easily fool yourself using the scientific method, if you don't keep questioning yourself and your work.

You are right but didn't go deep enough: you need interactive data. Not just static data. Environments.

You need a meaningful cost function

A cost function is more applicable in industry, less so in science. In science you're supposed to report what you find, to go wherever the findings take you.

Reply on news.ycombinator.com