Hacker News Reader

Comment by horsawlarway | Hacker News Reader

6 to 8 t/s is decent for a GPU-less build, but speaking from experience... it will feel slow. Usable - but slow.

Especially because these models <think> for a long bit before actually answering, so they generate for a longer period. (to be clear - I find the <think> section useful, but it also means waiting for more tokens)

Personally - I end up moving down to lower quality models (quants/less params) until I hit about 15 tokens/second. For chat, that seems to be the magical spot where I stop caring and it's "fast enough" to keep me engaged.

For inline code helpers (ex - copilot) you really need to be up near 30 tokens/second to make it feel fast enough to be helpful.

View on HN

Is a token a character or a word? I don't think I can read 30 words/second so 8/second would be fine. But if it's characters then yes, it would be slow.

Something in between. example: https://gpt-tokenizer.dev/

Looks like for the most part it's words, but some longer words get broken at prefix/suffix boundaries.

It typically somewhere between 3 to 5 characters.

It varies a lot by model, tokenizer, and language.

For DeekSeek 1 token ~= 3 English characters.

See: https://api-docs.deepseek.com/quick_start/token_usage/

It comes out to around 1-3 words/second. This is not so slow that it's maddening (ex 2 token/second is frustratingly slow, like walk away and make coffee while it's answering slow), but it's still slow enough to make it hard to functionally use and not break flow state. You get bored and distracted reading at that pace.

I've been with RooCode the slower token speeds are acceptable.

I'll ask it to do a task, then it will do it in a few steps and notify me when it's done. I use that time to take care of something else.

This is the same I get from a Core i5-9400 for smaller models. Is there no prosumer board that can take that much RAM? There must be a ThreadRipper that can do it, right? Why did he need EPYC?

It's not just the amount of RAM: RAM bandwidth also matters.

RAM channels is key here. That board has 24 channels of DDR5 RAM. A lot of lowend boards only have 2 channels. Most come in at 4. Some high end boards have 8, so 24 channels is quite wide.

Would DeepSeek-V3 be cheaper?

Only in the sense that it will generate fewer tokens.

Reply on news.ycombinator.com