HN
Hacker News
Top
New
Best
Ask
Show
Jobs
Comment by staticautomatic | Hacker News Reader
Parent
Full thread
staticautomatic
·
They’re called “lemma”
View on HN
Arnavion
·
The intent is the same, but as I understand it LLMs don't tokenize based on lemmas, though some of the tokens probably line up with them.
Reply on news.ycombinator.com