Hacker News Reader

> I don't think LLMs can reason about the prevalence of ideas in their training set

Good point. But isn't there a similar issue with things like 'now'? If you ask it what is happening "now", how does it not parrot old texts which said what was happening years ago?

View on HN

Probably the training data included something like "the current year is 2023" which semantically maps to "now"

When you say "semantically maps" do you mean that somebody somewhere coded such a "fact" into the training set? Or how is the mapping specified? If the training texts say "Current year is 2023" it would be wrong already :-)

More likely the model has a system prompt authoritatively saying what "now" is, and it can reason about other times specified in other resources in the training set because those resources specified their own time reference.

So even though a training resource said "It is DATE today. An IMPORTANT THING happened.", it knows that IMPORTANT THING happened in the past, because it knows CURRENT DATE from the system prompt, and it also knows that DATE < CURRENT DATE.

I see, So there is like a hi8gher level of facts/sources which are marked as such by the creators of the system. Higher level "facts" override the ones acquired from the public.

I didn't know about "system prompts".

Reply on news.ycombinator.com