Hacker News Reader

Comment by LoganDark | Hacker News Reader

That sucks. I wanted to save up for a couple years and get some hardware for home, but I guess the "AI" space moves so fast you barely get a couple months

View on HN

You would need ~250 groq cards to run a 7B model since their system doesn't scale down. So if you want to buy their hardware, you need a few millions dollars.

Their hardware was never for people at home, but for cloud providers.

That doesn't sound right. Their public demo ran on 568 LPUs because they had Mixtral-8x7B and LLaMA-70B (45B and 70B respectively). IIRC their cards each have slightly over 200MB of SRAM so this almost exactly checks out.

A 7B model would then be able to run on about 60 LPUs. Even at $20,000 per card that would be only $1.2 million and I highly doubt the cost is actually that high, that's just what DigiKey says the cost of an LPU is, if you're trying to buy just one :)

Save up for Tenstorrent instead.

I'll look into it, though seeing "contact us" always makes me think they're not going to sell a single unit to a home user. (With that said, Groq probably wouldn't either. You can technically buy LPUs for 20k each, without an expectation of support, but it takes tens of them to run Mixtral.)

Tenstorrent also looks incredibly Python-specific (as in, everything including their SMI seems mostly Python-based) which doesn't seem promising?

fwiw as a consumer I have a tenstorrent card in my machine

that is great. please hit me up privately, i'd love to chat with you about it.

Most of the low-level pieces are in Rust, the TUI is written in Python and most of the remaining pieces are getting lowered down to the Rust libraries over time.

(It was all Python up until ~6 months ago)

EDIT: Oh, and you can buy the Grayskull cards online now, without contacting anyone.

Even talking to people there, my experience is that they are super nice!

> EDIT: Oh, and you can buy the Grayskull cards online now, without contacting anyone.

I actually don't mind having to contact, I only mind if they won't want to sell to me due to being a non-bulk order.

> Most of the low-level pieces are in Rust

That's awesome!

I figured I'd add a bit of clarification here. Tenstorrent has a couple SW stacks. The lowest level programming (Metalium) model is written in C++ and goes down to the register level. Higher level entrypoints (ttlib, ttnn) are written on top of that in python. I think it's the smi tooling and other systems software that might be written in rust.

Reply on news.ycombinator.com