Is AI the new research scientist? Not so, according to a human-led study | Hacker News Reader

Is AI the new research scientist? Not so, according to a human-led study

news.warrington.ufl.edu

32 pointsgnabgib58 comments

It will never be. I don't know why people keep trying to make it into a research scientist. It's a great helper but it has no original insight and breakthroughs happen through original insight. LLMs are simply a conditional probability net of existing data so it can never ever have an original insight. I don't know why this is so hard.

And why is there even a desire to replace research scientists? Presumably this is the kind of job humans find meaningful and are good at. I don't understand AI as a replacement for humans instead of a smart tool for humans to make use of.

Why? to increase productivity and improve the human condition. If AI can do research then technological and scientific progress will increase dramatically.

There's an unknown cost if all human endeavor becomes replaceable by AI. I would be cautious about this.

For who though? The research scientist can no longer do the work they trained for. Apply this to the entire economy at some point, and you have a serious problem.

In a world where human labor is no longer necessary, why do you think the people who control the AI models will care about “improving the condition” of anyone but themselves?

Human labor hasn't been necessary for a large portion of people. We have more than enough people to make food and collect water for us to survive. We can always make up jobs for people to do.

Talk to the person who pays the scientist. Anyone who works, works for someone. Who you work for is the person who wants to replace you.

I don't see that as a net good for society. I do see it as eventual grounds for another French-style Revolution.

Society doesn't work because everyone is collaborating for the greater good. Society works because everyone generally cares very little about society and they care more about themselves. The net effect of everyone caring only about themselves is the result you have today.

That's a very western, libertarian way of looking at things. But even so, replacing workers en masse leads to high unemployment and civil unrest. If we get to the point of automating research scientists, then a whole lot of jobs can be automated.

Most of society and people on HN are from western society. I'd say it's more of a capitalist view point. Which is basically all of society you see today.

To ensure net benefit to society one must make policies that benefit individuals and hope that the network effects balance out and make society better overall. Expecting altruism is unrealistic.

Or Luddites destroying machines that replace them while using machines as tools to make their labor easier.

The working class wants to automate labor to increase their free time.

The owner class wants to automate labor so they can rid themselves of the working class.

Why is there even a desire to replace software developers? Presumably this is the kind of job humans find meaningful

Why is there even a desire to replace car manufacturers? Presumably this is the kind of job humans find meaningful

[...]

Reduce labor costs to increase profits. But that's for the good of a small number of people when all meaningful work can be automated. And I don't trust the billionaires to make society better for the rest of us.

I wouldn’t be so dismissive. Research is just a loop of hypothesis, experiments, collect data, make new hypothesis. There’s so creativity required for scientific breakthroughs, but 99.9% percent of scientists don’t need this creativity. Just need grit and stamina.

I think this comment is significantly more dismissive of science and scientists than the original comment was of AI.

And insight. Insight can be gleaned from a comprehensive knowledge of all previous trials and the pattern that emerges. But the big insights can also be simple random attempts people make because they dont know something is impossible. While AI _may_ be capable of the first type, it certainly won't be capable of the second

Awfully bold to claim that 99.9% of scientists lack the need for "creativity". Creativity in methodology creates gigantic leaps away from reliance on grit and stamina.

I wouldn't be so dismissive of the objection.

That loop involves way more flexible goal oriented attention, more intrinsic/implicit understanding of plausible cause and effect based on context, and more novel idea creation than it seems.

You can only brute force things with combinatorics and probabilities that have been well mapped via human attention, as piggy-backing off of lots of human digested data is just a clever way of avoiding those issues. Research is by definition novel human attention directed at a given area, so it can't benefit from that strategy in the same way domains which have already had a lot of human attention can.

I think the whole idea of "original insight" is doing a lot of heavy lifting here.

Most innovative is derivative, either from observation or cross application. People aren't sitting in isolation chambers their whole lives and coming up with things in the absence of input.

I don't know why people think a model would have to manifest a theory absence of input.

> I think the whole idea of "original insight" is doing a lot of heavy lifting here.

This is by biggest issue with AI conversations. Terms like "original insight" are just not rigorous enough to have a meaningful discussion about. Any example an LLM produces can be said to be not original enough and conversely you could imagine trivial types of originality that simple algorithms could simulate (i.e. speculate on which existing drugs could be used to treat known conditions). Given the amount of drugs and conditions you are bound to propose some original combination.

People usually end up just talking past each other.

The biggest problem with LLMs isn't that it lacks original insight. It's that the insight is so original that we call that insight hallucinations.

We like to think Humans are the most creative things on the face of the earth and we don't like to attribute creativity to LLMs. The sad reality is that LLMs are likely more creative then humans.

Most humans are also too creative, but we have moderating impulses that tell us so much. Very few humans have the skill of being able to ride the cutting edge without going too far off either side of it, and most can only do that in a very narrow subfield.

I think the distinction is that hallucinations are incorrect. You can be super creative building a new chair, but if you can’t sit in it, it’s not a chair.

Right. So you have a testing framework/agent/other llm. It’s not like our brain is one independent machine. It’s various parts all contributing different aspects of intelligence.

This makes no sense. You can describe the brain reductively enough and make it sound like it can't have an original insight either. Transformers are expressive enough function approximaters in theory, there's no reason why a future one couldn't have novel insights.

This is such a weird misconception I keep seeing - the fact that the loss function during training is minimising CE/maximizing prob of correct token doesn't mean that it can't do "real" thinking. If circuitry doing "real" thinking is the best solution found by SGD then it obviously will

you're conflating LLMs with AI. Strictly speaking, from what we understand of physics, chemistry, biology, computing, and mathematics, there is no coherent argument that you could not build an AI which could be an effective research scientist (which I define as: a system which produces novel hypotheses based on current scientific knowledge that are likely to be true, and is capable of evaluating their likelihood quickly enough to be relevant to human endeavors.)

I imagine that such a system would probably have at least required component that looked much like an LLM.

Not all breakthroughs happened due to original insight- many came from tediously improving techniques through fairly mundane means, or from advancements in other areas.

> which I define as: a system which produces novel hypotheses based on current scientific knowledge that are likely to be true, and is capable of evaluating their likelihood quickly enough to be relevant to human endeavors.

Produces hypotheses which are likely to be true? Pardon my ignorance, have we even proven gravity to be true yet? Sure, I think gravity exists and is true, however your definition of AI seems like Swiss cheese.

I think you must have misunderstood my statement on multiple levels (or you're being disingenuous; it's hard to tell). I don't think you can ever prove anything to be true in science, and I expressed that in my statement ("likely to be true"). You could also turn what I said around, and say "novel hypotheses which are unlikely to be trivially falsified".

There's nothing swiss cheese about my heuristic definition of a research scientist AI; it's precisely the same thing that we expect human research scientists to do (I presume an AI research scientist could also write papers that get published and grants that get approved, but unlikely to be an effective mentor for a PhD candidate). It's also only a working definition that I would update if I found a good reason.

Sorry, I did not mean to be disingenuous. My skepticism came through in my tone. I think LLMs are a valuable tool, and will continue to grow in different ways. I can even imagine a scenario where the right series of prompts gives someone a eureka moment in a big, new way.

I don’t see how an LLM could produce anything novel without being prompted. At which point, is the LLM just a tool for a brilliant mind, or is the LLM the brilliant mind? I prefer the former, because I just can’t wrap my head around the latter.

Why do you keep talking about LLMs? An AI research scientist would almost certainly contain many more components.

Because it’s not AI.

Yeah, that's exactly what a HUMAN would say ...

Sounds like they only evaluated GPT-4o and weaker LLMs like mid-last year?

I was thinking last night. Shouldn't software we make help people instead of replacing them? Why must innovation be in the direction of and at the cost of replacing humans?

We've been optimizing for the wrong metrics? Infinite growth was fine when the map had "here be dragons", led to absurdum you get the profit driven, neurotic company architecture that optimizes for the goals of optimizing, every single person can be playing the right cards but the end goal is to move value from A to B with A(A-value) being not considered, when A(-value) are people with lives, we either pickup and move, but A_n(-value) is already there. Aka. no more dragons.\

edit for clarity

That's what it does.

"Replacement" is only a problem for people who are dependent on someone else being dependent on them.

> "Replacement" is only a problem for people who are dependent on someone else being dependent on them.

Not so. Replacement is a huge problem for people who have people who depend on them to furnish the cost of living.

Also it can be quite dangerous in a game setting where some costs of losing the game include homelessness or death.

In fact, it might be desirable to some political figures to drive up enlistment numbers by putting more people in such precarious situations.

But what do I know, I read a book and an AI can do that for you now... so... don't think too much about it.

>Also it can be quite dangerous in a game setting where some costs of losing the game include homelessness or death.

It could allow for natural selection to start taking place again to select for desirable traits.

A meteor could select for the whole Earth to be gone, but I don't think I'm going to start praising disaster.

It doesn't have to be. But often the executives or investors who stand to profit the most from innovation also have strong public facing influence over the narrative. Employees cost a ton, so it's self serving both to promote the product to like minded people, and to hype the product itself.

"I was thinking last night. Shouldn't <all innovation> we make help people instead of replacing them? Why must innovation be in the direction of and at the cost of replacing humans?"

-Humans when electricity replaced lamplighter jobs [1]

[1]: https://sloanreview.mit.edu/article/learning-from-automation...

True then, still true now.

I really don't care if jobs are replaced, so long that people are still able to make a living.

It really becomes a problem if you replace humans as a whole, and don't come up with something to allow them to make a living still such as UBI or others.

I think that is the big difference between the lamplighter situation, and the situation at hand.

I've seen no evidence for there being "not enough jobs". The problem (as always) is the status associated with the job.

There is a real dearth of people wanting to work the trades, but no one wants those jobs. I'm not sure how to solve that problem, even I don't want that job.

A smart society would give status to people who take jobs that are needed but undesirable. We do not live in a smart society.

Related is the fact that other jobs existing, even IF desirable, doesn't mean they can be learned over night. Many jobs worth having take years of training. Switching careers overnight is not realistic for nearly anyone who gets laid off due to replacement.

That seems like a transitory problem already solved by government programs such as extended unemployment and adult training programs.

The problem is still more people's willingness to cede the status of their current jobs to take up new ones. That's still the hardest part.

I'd argue that your point is only true if the amount of money the government gives out per month per person vastly increases.

Yup. Political problems irrelevant to the tech. Also a lot more attainable/manageable than an expansive permanent UBI.

That is a quite pervasive thought in the tech industry nowadays: we break everything, let the politics/society/aliens fix it. What would you say if the politics/society/aliens decide to block said tech, stopping the problem at the source? It's not a given, that techbros can do anything with impunity.

Another headline to correct: "whiney desperate scientists fearing their grift is up, try to claim that AI research isnt that good"

How can we trust a human to run the study, isn't there a bias? Needs an AI prompted to be a research scientist as a co-author for it to be balanced.

.. and if that AI does not give you the answer you want, re-run it with minor prompt modifications.

People are really over indexing on current AI capabilities.

We’re barely 2 years on from ChatGPT’s initial release and we’ve gone from “this thing can put words together in a semi-coherent way” to “this thing produces undergrad level research papers on anything you ask about”.

Where will we be in another 2 years? Probably not at AGI, but there’s no sign this is slowing down.

I dunno, I remember reading qbout glue on pizza almost a year ago.. and today I was talking to github tech support and their AI bot (presumably latest and greatest, with best minds programming it), suggested a command which does not exist. And Google AI summary is still hilariously bad for any moderately complex question.

I don't see much AI yielding accurate answers anytime soon, and certainly not in 2 years.

The best models are not GitHub’s support bot (Microsoft isn’t even creating their own models) or Google’s AI summary.

If you haven’t used Claude 3.7 extended thinking to write code or ChatGPT Deep Research to investigate a topic you are not seeing what the capabilities are at the cutting edge.

https://aider.chat/docs/leaderboards/

None of it is perfect, obviously, and it’s not going to take everyone’s job next year. But people are not updating their thinking properly if they haven’t used the latest paid models.

Reply on news.ycombinator.com