Hacker News Reader

Comment by gzread | Hacker News Reader

By emitting a next token distribution with a 10% chance of 0, 10% chance of 1, etc.

Also it's an LLM, not a brain.

Interesting. So you expect it to "not think" and simply produce a value corresponding to "it's the same to me", knowing that it will be translated into an actual random value.

Instead, exactly as a person would do, it does think of a specific number that feels random in that particular moment.

If I care a little bit about that random number I might reach for my phone and look at the digits of the seconds of the current time. It's 31 now. Not appropriate for multiple lookups.

Yes, there is probably some variable context in every chat (like date and time). Could work as a good seed but I guess you should ask the LLM to really make an effort to produce a seriously random number. (Actually I've just tried, even if you ask it to make an effort, the number will be always the same).

If "a random number is" is followed by all digits equally in the training dataset, then emitting a uniform distribution should minimise cross-entropy, right?

Er, should it? Even of you trained an LLM exactly over this type of question, if the sequence to predict in the training data is really random, then any output is equally wrong. Even if the output is a fixed "1234" or "0000". There is no signal to train on, not even one that favours an equal distribution.

On the other hand, LLMs show they know very well what a random number is and the fact it just shouldn't look like anything in particular, so they strive to come up with a number that doesn't look like anything in particular. Which happens to be always the same number given the starting conditions.

No LLMs are calibrated?

What?

Reply on news.ycombinator.com