Hacker News Reader

Comment by iandanforth | Hacker News Reader

Sounds like you might be more the target for the $3k 128GB DIGITS machine.

Weirdly they're advertising "1 petaflop of AI performance at FP4 precision" [1] when they're advertising the 5090 [2] as having 3352 "AI TOPS" (presumably equivalent to "3 petaflops at FP4 precision"). The closest graphics card they're selling is the 5070 with a GPU performing at 988 "AI TOPS" [2]....

[1] https://nvidianews.nvidia.com/news/nvidia-puts-grace-blackwe...

[2] https://www.nvidia.com/en-us/geforce/graphics-cards/50-serie...

It seems like they're entirely different units?

TOPS means Tera Operations Per Second.

Petaflops means Peta Floating Point Operations Per Second.

... which, at least to my uneducated mind here, doesn't sound comparable.

I'm assuming that the operations being referred to in both cases are fp4 floating point operations. Mostly because

1. That's used for AI, so it's plausibly what they mean by "AI OPS"

2. It's generally a safe bet that the marketing numbers NVIDIA gives you is going to be for the fastest operations on the computers, and that those are the same for both computers when they're based on the same architecture.

Other than that, Terra is 10^12, Peta is 10^15, so 3352 Tera ops is 3.352 Peta ops and so on.

I’m really curious what training is going to be like on it, though. If it’s good, then absolutely! :)

But it seems more aimed at inference from what I’ve read?

I was wondering the same thing. Training is much more memory-intensive so the usual low memory of consumer GPUs is a big issue. But with 128GB of unified memory the Digits machine seems promising. I bet there are some other limitations that make training not viable on it.

It will only have 1/40 performance of BH200, so really not enough for training.

Primarily concerned about the memory bandwidth for training.

Though I think I've been able to max out my M2 when using the MacBook's integrated memory with MLX, so maybe that won't be an issue.

Training is compute bound, not memory bandwidth bound. That is how Cerebras is able to do training with external DRAM that only has 150GB/sec memory bandwidth.

The architectures really aren't comparable. The Cerebras WSE has fairly low DRAM bandwidth, but it has a huge amount of on-die SRAM.

https://www.hc34.hotchips.org/assets/program/conference/day2...

They are training models that need terabytes of RAM with only 150GB/sec of memory bandwidth. That is compute bound. If you think it is memory bandwidth bound, please explain the algorithms and how they are memory bandwidth bound.

Reply on news.ycombinator.com