Google Co-Scientist AI fed previous paper with the answer in it | Hacker News Reader

Google Co-Scientist AI fed previous paper with the answer in it

pivot-to-ai.com

200 pointspcfwik36 comments

Open articleView on HN

> Prof Penadés' said the tool had in fact done more than successfully replicating his research.

> "It's not just that the top hypothesis they provide was the right one," he said.

> "It's that they provide another four, and all of them made sense.

> "And for one of them, we never thought about it, and we're now working on that."

The Google's co-scientist still seems to make a useful assistant.

Colour me unsurprised - even not knowing data leakage had occurred, the hypothesis was underwhelming, as I mentioned in a comment on an earlier discussion. I sometimes despair for the state of thinking in science these days given how quickly people fawn over entirely pedestrian thinking and work.

https://news.ycombinator.com/item?id=43105759

If the AI found a needle in a haystack, that isn't bad, is it?

The point isn't that it's bad, the point is that the AI did not discover the solution to the problem independently, as news headlines had implied.

it is if you're presenting it as coming up with novel information rather than essentially acting as a search engine for information from its training data (which we already know ai does well, at least with some caveats)

Do you think it's possible that the person who literally wrote (multiple) papers with the answer AI gave, may phrase their question to the AI with similar, or even the exact phrasing they used in previous papers on the same subject ? If I load a bunch of lorem ipsum into AI scientist, labeled as scientific papers, then ask the AI "lorem ipsum" do you think it'll spit out some gibberish from those specific papers?

Instant subscribe when I saw David Gerard's name. He cut through the BS in crypto and we need more people like him focusing on AI fraud like this.

thank you!

I mean, to be fair, "knowing all of the relevant literature" is a great first step for solving a problem! And a LLM can probably be better about not forgetting than humans are (though I'm guessing they do much worse on the hallucination front than humans do- humans tend to know when they are playing a hunch). But "This paper from a lower-tier journal two years ago suggests you should look at X" is a very valuable thing to have!

not surprising. why would you expect a simple next token predictor to think?

Technically everyone is a next token predictor. The only difference is how good the prediction is.

In humans, language is based on the state of the world, and anticipates future states of the world. In an LLM, language is just language.

If an LLM says "my favorite flavor of ice cream is cookie dough", it's just a statistically plausible thing to say.

If I say "my favorite flavor if ice cream is cookie dough", it's because I'm trying to deceive someone, because for some reason I don't want them to know my real favorite flavor is cake batter.

You haven't identified any difference. In order to generate a really good statistically plausible thing to say you need a model of the state of the world and anticipated future states. This is very well known. I wish we didn't need to keep going over this basic stuff, but it somehow comes up again and again every time AI is discussed here.

Of a world, not necessarily the real world. One of the main drawbacks of philosophical rationalism is a mind's ability do deceive itself with constructs that sound plausible and are consistent with the world model. A healthy dose of empiricism is the best cure discovered so far.

In other words, less thinking and more experiments and observations. Thinking alone leads to ignorance.

No, a model of the real world. Not necessarily an accurate no model.

> everyone is a next token predictor

As true as "everyone is a calculator". There are superficial similarities, but that we achieve a similar result does not mean that we follow the same reasoning (or lack of).

> does not mean that we follow the same reasoning

Finally some logic. Token prediction alone - the behavior - isn't proof of a lack of intelligence.

Better reasoning gives better token predictions. And, sometimes, the ability to invent new tokens entirely.

This is a deepity.

The trivial interpretation is: every word written can be constructed by optimizing a prediction based on current state, what has been written so far, and a sufficiently complex model. This is true of anything computable: just make the method implicitly contain the program by assigning a high probability to any token that is consistent with running the computation one more step. It's also true of anything expressible: just brute force a solution that can be expressed in n words, then assign a high probability to the first word of these n words.

The profound but wrong interpretation is that intelligence is just statistical prediction according to some general-purpose algorithm, and that this algorithm is tractable. Consider something like solving a SAT problem. You're going to have a hard time using any tractable general-purpose algorithm to predict whether x_2 is true for the satisfying solution based on some long CNF statement plus "x_1 is false".

Now, what you _can_ do is augment your model so that if the previous tokens constitute a CNF-SAT instance plus a partial answer, then you cart these off to a SAT solver and output its next token. But the more you do this, the less force the "mere statistical prediction" part holds. The "next-token predictor" is just an interface to an assembly of different approaches; and often, these approaches (like the SAT solver) will output the whole solution all at once for free.

Animals are not next token predictors. Most animals don't have language and don't generate sentences/ideas. It is unique feature of humans and big part of our success.

Animals do have sensors and can reason about real world quite intelligently. This is what LLM- models don't have. Surely, all the text in Internet has a lot of textual information about real world, but something "obvious to animal" might not be there.

I feel us humans are kind of both: We have sensors and experiences about real world, but also oral and written tradition.

Once we develope LLM- with cabability to sense and experience real word (perhaps with some evolutionary algorithm to make it better over time) it should start to be close to human beings.

Not true, we build novel knowledge all the time. You only have to look at *gestures at everything *.

We would not generate novel knowledge if our only interface with the world was a short text based conversation.

In other words, I don't think that's a fundamental difference between humans and AI; we just haven't given AI any opportunity to build novel knowledge.

Part of that is because it can't really interact with the world yet, in a significant way. Part of it is because nobody has cracked on-line learning.

Actually when you give AI the ability to interact with a world, as in RL, it does build novel knowledge.

Right, sometimes we invent new tokens. It's unclear whether large models can do that yet.

Technically, everyone is just an ugly giant bag of mostly water.

This is incredibly wrong.

I mean the logic behind it is sound. Just infantile compared to what we do. But yes I believe that all we do boils down to "if x is true and also y is true, therefore under these circumstances z must be false, I'll give it a go and find out". Plus a little qualified randomness, look at the large percentage of human discoveries being made accidentally (albeit by qualified professionals making best guesses).

Please rigorously define what the term "next token predictor" means.

Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More formally:

  P(tₖ|t₁,..,tₖ₋₁)

For example, the original OpenAI GPT tranformer model was trained to optimise the objective:

  ∑ log P(tₖ|tₖ₋ₙ,..,tₖ₋₁;Θ)

Where n the length of a sliding "context" window and Θ the parameters of a neural net model (here, a Transformer architecture) used to optimise the objectivbe.

See page 3 of the OpenAI paper:

https://cdn.openai.com/research-covers/language-unsupervised...

Also see, for alternative formulations:

Hidden Markov Models:

https://en.wikipedia.org/wiki/Hidden_Markov_model

N-Gram word models:

https://en.wikipedia.org/wiki/Word_n-gram_language_model

> Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More formally:

> P(tₖ|t₁,..,tₖ₋₁)

Not a meaningful definition unless you say what probability distribution it's maximizing it for!

Because for any system that takes any actions (e.g. human beings), there's some probability distribution that that's true for...

The probability distribution that is being modelled is the conditional distribution P(tₖ|t₁,..,tₖ₋₁).

That’s not a description of a probability distribution.

It is.

No, the conditional distribution is a construction that transforms one probability distribution to another; you still haven’t said what the underlying distribution was.

Nope, there are no numbers there, how can you put in a position and get a value (probability) out? Saying "it is" is not evidence or an explanation.

Here is the normal distribution probability function.

https://en.wikipedia.org/wiki/Normal_distribution

You can see that there is an actual equation where all the numbers can be filled in except for X, which is the function input.

A device or system that, given a set of tokens, predicts the next token purely based on the previous tokens and its own initial state. It is incapable of coming up with a new token type or changing its own workings

On top of your definition being a bit circular, it is also like saying humans can't come up with new things, because we can't come up with new characters when talking online. We're limited by the character-set we're given but that doesn't mean we can't conceive of new characters and more importantly, can't represent new characters using existing character-sets using some encoding scheme to represent that new character. To that end, lets check with chatGPT if it can come up with a new and useful concept for a token, using existing characters to encode for it.

https://chatgpt.com/share/67bd04cf-1c50-8008-8606-ccac5df989...

And there you have it, novel constructed from available ones, with a genuine pragmatic and LLM specific use case.

So if they can come up with their own internal representations is that sufficient?

https://arxiv.org/abs/2410.05864

Or are you wanting them to use keys that are not on their keyboard?

Reply on news.ycombinator.com