Comment by born1989 | Hacker News Reader

Top New Best Ask Show Jobs

Comment by born1989 | Hacker News Reader

Parent Full thread

Thanks! Isn’t “an Astronomer” a single word for the purpose of answering that question?

Following your comment, I asked “Give me pairs of synonyms where the last letter in the first is the first letter of the second”

Claude 3.7 failed miserably. Chat GPT 4o was much better but not good

Don't know about Claude, but at least with ChatGPT's tokenizer, it's 3 "words" (An| astronom|er).

That is a sub-token task, something I'd expect current models to struggle with given how they view the world in word / word fragment tokens rather than single characters.

"An astronomer" is two tokens, which is the relevant concern when people worry about this.

Reply on news.ycombinator.com