Hacker News Reader

Here is the list of the 100k GPT-4 tokens as text file.

Yes, a lot of tokens are just for code.

Edit: Here as raw link for the poor mobile devices:

Where are all the tokens for other languages though? How are these getting tokenized?

Thanks for this! That was very nice and thoughtful.

There’s something poetic about ULL being a token, but NULL not being one.

Saw many words missing their first letter. Realized it’s probably because it’s sometimes Null and sometimes null

ULL is also used for designating Unsigned Long Long 64bit ints in c++, so it’s not just a part of the null symbol.