Hacker News Reader

Could someone please share a good resource for building a machine from scratch, for doing simple-ish training and running open-source models like Llama? I'd love to run some of these and even train them from scratch, and I'd love to use that as an excuse to drop $5k on a new machine...

Would love to run a bunch of models on the machine without dripping $$ to OpenAI, Modal or other providers...

View on HN

I am no where near an expert on this subject, and this information is from a few months ago so maybe it's outdated, but people on Reddit[1] are claiming running the llama with 65B parameters would need like 20K+ of GPUs. A 40GB A100 looks like it's almost $8K on Amazon, and I'm sure you could do a lot with just one of those, but that's already beyond your $5K budget.

[1] https://www.reddit.com/r/MachineLearning/comments/11i4olx/d_...

I'll let others chime in but you could still probably build something really powerful within your budget that is able to run various AI tasks.

You can get around 4-5 tokens per second on the 65B LLaMA with a 32 core 256GB ram Ryzen CPU, not sure how much it costs to build but can rent one from Hetzner for around two hundred bucks a month.

There are some threads with hardware recommendations in the LocalLLaMa subreddit.

Here’s a recent one:

https://www.reddit.com/r/LocalLLaMA/comments/13f5gwn/home_ll...

Reply on news.ycombinator.com