Comment by hn_throwaway_99 | Hacker News Reader

Comment by hn_throwaway_99 | Hacker News Reader

Parent Full thread

hn_throwaway_99

Why do you find it fascinating? I have the most rudimentary understanding of LLMs and it seems to me the least fascinating thing about LLM limitations.

That is, all LLMs look at language as a series of opaque, independent tokens, e.g. strawberry might be a single token (say <5678>), or probably two (e.g. straw and berry, say <123><789>). But in no way will it represent it like we will, with letters. So if you ask it "how many R's are in strawberry", it fundamentally can't do any "counting", it just basically has to rely on whether that question (or similar questions about related words) has been asked before so it can predict the next token in its output correctly. I suppose with enough data and LLM could learn to associate character counts with tokens (e.g. with the right training set it could learn metadata about the tokens).

My point being that with only the most basic understanding of how LLMs function, this "character counting" limitation seems blatantly obvious.

I don’t think we’ve yet come to the point where, how an LLM end to end goes from prompt to output is blatantly obvious.

LLMs operate with language at multiple levels of abstraction and tokens are not the only way to have character level knowledge.

For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge.

And of course LLMs famously have emergent properties, for which there’s not yet a precise mechanism to illuminate the results.

We’re dealing with very complex systems that are still relatively poorly understood, and I believe the pool of concepts understood to the point of being blatantly obvious is still small.

The fact that large parts of how LLMs work is poorly understood doesn't mean that that we should brush the whole thing with "it's a complex mystery!"

We know exactly how LLMs tokenize their input - OpenAI even has a public tool you can test out, https://platform.openai.com/tokenizer .

When you say "For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge." - yes, that's why I explicitly said "e.g. with the right training set it could learn metadata about the tokens".

But the point was still put perfectly by another commenter: "How many 1 bits are there in strawberry?" When nearly all humans can't answer that, we're very unsurprised; we see it immediately as a simple difference in how humans encode that string vs. how computers do it. We don't say "Well, the human mind is so complex and filled with emergent properties that the reason for this human limitation is a big mystery". And we also know that if we teach a human the encoding rules for characters (i.e. the bit pattern of each letter), they could answer this question, and similarly that's the analogous thing to training an LLM to learn metadata about the tokens.

Now, what I think is very interesting is why it's so hard to teach an LLM to say "I don't know" when asked to count characters. In my opinion, that's a much more interesting limitation that gets at some of the root, fundamental differences in how LLMs function.

teaching LLM to say 'i don't know' in general is hard, not just for counting letters.

It can't without some Bayesian action.

You seem pretty certain for having only the most rudimentary understanding.

I’m still going to have to disagree. I’d describe the idea that token granularity is the cause of letter counting limitations as a hypothesis, not as something that’s been conclusively shown as far as I’m aware.

I’m not discounting it, or even saying it’s unlikely, but its not hard to imagine other ways it could plausibly be happening.

As a side note when I mentioned training data I was not suggesting anything related to “token metadata”, or to tokens in any regard. Rather, I think it might be possible to instead improve the learning around certain types of counting in a way that could generalize.

> You seem pretty certain for having only the most rudimentary understanding.

Fine, I lied. I have a much better understanding than you. Happy?

I only wish you the best brother.

Reply on news.ycombinator.com