HN
Hacker News
Top
New
Best
Ask
Show
Jobs
Comment by neoromantique | Hacker News Reader
Parent
Full thread
neoromantique
·
Something in between. example:
https://gpt-tokenizer.dev/
View on HN
UncleOxidant
·
Looks like for the most part it's words, but some longer words get broken at prefix/suffix boundaries.
Reply on news.ycombinator.com