Hacker News Reader

I'm gpu poor so I use cpu to run large models. For example, llamafile running mixtral 8x22b on my threadripper can chew through long legal documents and give me advice in a few minutes, using f16 weights and kahan summation. Show me someone who has that many graphics cards.

View on HN

Are you "gpu poor" by choice? I would have thought you, of all people, would be "gpu wealthy" like few can imagine.

I'm a hobbyist developer. I haven't been employed in six years. In the past few months I've been fortunate enough to find work as a TVC for Mozilla, which has been wonderful. It's a good lifestyle that affords me a great deal of freedom and autonomy, but certainly not a lot of GPUs. I have an RTX 4090, a few XTX cards, and I rent stuff from Vast occasionally, but that's about it.

I love how RTX 4090 is now "GPU poor"

It can't run most of the open source models being released today. It's not useful if your job is to keep up with LLM releases.

What are your tokens/s on that setup?

Reply on news.ycombinator.com