Hacker News Reader

Comment by ascorbic | Hacker News Reader

The o1 example is interesting. In the CoT summary it acknowledges that the most recent official information is 1611m, but it then chooses to say 1622 because it's more commonly cited. It's like it over-thinks itself into the wrong answer.

View on HN

Does it search the internet for that? I assume so because else claiming how often something is cited does not make sense, but would be interesting to know surely. Even gpt4o mini with kagi gets it right with search enabled (and wrong without search enabled - tried over a few times to make sure).

I don’t think the public o1 can search the internet yet, unlike 4o. In principle it could know that something is more commonly cited based on its training data. But it could also just be hallucinating.

> it could know that something is more commonly cited based on its training data

No there is no such concept or way to do something like that. LLMs do not have such kind of meta-knowledge over their training data or weights. But there could be explicit mentions about this on their training data and they could pick on that and that is probably the simplest explanation.

>LLMs do not have such kind of meta-knowledge over their training data or weights.

Not sure this is a claim that can be confidently made.

https://arxiv.org/abs/2309.00667

https://x.com/flowersslop/status/1873115669568311727?t=eBMbK...

> In principle it could know that something is more commonly cited based on its training data

Could it? Without explicit training for that, how would it be expected to know it has to be able to count occurrences of something?

I think it would be more vibes based - commonly occurring things would be reinforced more in the weights. Rather than it explicitly counting the number of occurrences.

So the probabilities would be skewed towards something, but unless the model could somehow count/infer its own weights, I don't see how it could "introspect" to see if something is more common than something else.

Can the claim about citation frequency be just an answer pattern and not model's exact reasoning?

Yeah there could be parts of the training set with 1611 being explicitly called the official and 1622 being explicitly called the most common answer. But it can also have access to search results directly I think. Is there a way to know if it does or not?

I think i had similar case yesterday for Python script. It gave me code for older version of a module, but when i pasted the error i got, it corrected itself and gave me proper solution for version i had installed.

How could a language model infer that the official information overrules anything else?

Same way as we can: learning which sources are more trustworthy.

There's limits to how far you can go with this — not only do humans make mistakes with this, but even in the abstract theoretical it can never be perfect: https://en.wikipedia.org/wiki/Münchhausen_trilemma — but it is still the "how".

for the last 25+ years we rather not learned, but trusted the top3 of SERPs. Every ranking algorithm will be gamed eventually

I would say that we learned to trust the search engines; but otherwise I agree with you: every ranking algorithm will be gamed eventually.

(I wonder if giving an LLM content with intent to cause its users to spend money they didn't need to, would count as fraud, hacking, both, something else entirely?)

I’m not sure what kind of response you’re looking for, or if this is a rhetorical question or not. But “how could a language model infer…?” can be asked about a whole lot of things that language models have no problem reliably inferring.

> that language models have no problem reliably inferring

... the article did give me a different impression.

I don’t think you read my comment correctly.

Attention models learn what to pay attention to.

It's been found that data that begin with "Wikipedia:" are automatically weighted higher by language models during training, completely unsupervised.

But this is the same problem - Wikipedia is a secondary source and should always get overruled by the primary source.

And there are documented cases of concentrated efforts to manipulate the content of Wikipedia.

You’re probably being facetious but just in case:

https://www.bbc.com/news/technology-28481876.amp

https://www.bbc.com/news/technology-58559412.amp

(With apologies for amp links)

I was most certainly not being facetious.

It's still better than the information on most of the internet, which is what most of the dataset is.

Reply on news.ycombinator.com