Hacker News Reader

Comment by thom | Hacker News Reader

There appear to be slightly weird commercial reasons behind this, because gaming GPUs have great CUDA performance but NVIDIA won’t let you put them in a datacentre. So buying your data scientists gaming laptops (RGB and all) generally works out faster for any reasonable price point. That said, a dedicated server with a decent Xeon and MKL set up correctly generally outperforms CPU-bound stuff.

View on HN

I think it really depends on your data size. All the benchmarks I can find are on massive datasets, with tens of millions of rows or thousands of columns. I’m sure there are significant performance gains in these situations. Our data just wasn’t big enough.

There was a time when I thought 10s of millions of rows was massive. Now it just send run of mill.

With larger data it really depends on the algorithm. If you must iterate over more than a few GB at a time, GPU memory capacity and bus speeds become prohibitive, while a dead-simple implementation on a single CPU with 100+ cores and TBs of RAM goes brrr.

CPU RAM is generally much slower.

8 channels of DDR4-3200 only provide 200GB/s bandwidth. RTX 3090 has 936GB/s. So even 4 socket Xeon won't catch up.

That would only be an advantage if you had to do multiple passes over the data, otherwise the data would still go through the CPU RAM before getting loaded onto the GPU, no?

Definitely the case in the state of the art stuff like neural networks.

Many if not most other algorithms are iterative. Hell, even sorting is.

When the models get sufficiently big, even a 40GB A100 is not sufficient. Unless you can feed the core quick enough, your performance drops considerably.

GPUs are like heavy flywheels. Getting them up to speed takes some time (copy data, compile and copy the kernels, kickstart everything, etc.), so you need to start them once to get the performance benefits. Otherwise CPU is much more nimble since they're closer to RAM and made to juggle things around.

What prevents gaming GPUs from being used in a data center? Is there some licensing restriction?

Yes, it’s part of the EULA for the driver.

> The updated end-user license agreement (EULA) states: “No Datacenter Deployment. The software is not licensed for datacenter deployment, except that blockchain processing in a datacenter is permitted.” [0]

I guess it's time to invent a blockchain that trains ML models as PoW :)

As a sidenote, I have rented servers with GeForce cards from multiple providers in multiple countries, so this rule doesn't seem to be respected very much. And since it's part of the driver EULA, nvidia can't legally go after server providers, since they don't install any drivers, just build and rent out the hardware. For all they know, all their customers are running noveau.

[0] https://www.datacenterdynamics.com/en/news/nvidia-updates-ge...

Reply on news.ycombinator.com