So tokens aren’t as important.
I got 0.863 (for 1st)/0.559 (for 2nd)/0.447 (for 3rd) accuracy for Qwen 3 8B model embeddings. Note the code is hacky and might be wrong in ways + in reality transformers do know more because here I utilize only embedding layer. However it does show there are very clear signals on characters in tokens in embedding vectors.
I wonder if it would help to explicitly insert this info into an embedding vector, similar to how we encode word position info. For example, allocate the first 20 vector elements to represent ASCII codes of token's characters (in some normalized way).
But to be frank I don’t think it’s really needed, I bet everything really needed model learns by itself. If I had time I would’ve tried it though :)
Bonus content, accuracies for other models (notice DeepSeek!):
- Qwen3-32B: 0.873 / 0.585 / 0.467
- Qwen3-235B-A22B: 0.857 / 0.607 / 0.502
- DeepSeek-V3: 0.869 / 0.738 / 0.624
I took Qwen3 1.7B model and did the same but rather then using embedding vector I used vector after 1st/etc layer, below accuracies for 1st positions:
- embeddings: 0.855
- 1st: 0.913
- 2nd: 0.870
- 3rd: 0.671
- 16th: 0.676
- 20th: 0.683
And now mega bonus content: the same but with prefix "count letters in ":
- 1st: 0.922
- 2nd: 0.924
- 3rd: 0.920
- 16th: 0.877
- 20th: 0.895
And for 2nd letter:
- embeddings: 0.686
- 1st: 0.679
- 2nd: 0.682
- 3rd: 0.674
- 16th: 0.572
So "the sky is blue" converts to the tokens [1820, 13180, 374, 6437]
And "le ciel est bleu" converts to the tokens [273, 12088, 301, 1826, 12704, 84]
Then the embeddings vectors created from these are very similar, despite the letters having very little in common.
This is incorrect.
strawberry is actually 4 tokens (at least for GPT but most LLM are similar).
To the extent the knowledge is there it’s from data in the input corpus, not direct examination of the text or tokens in the prompt.