Hacker News Reader

Comment by andrewstuart | Hacker News Reader

In the tests I’ve done, retail GPUs are much faster than data center GPUs.

They are, but data center GPU (ampere and above) are twice as good in performance per watt.

Aside from being green and that, why would a consumer care about a leased processor's performance per watt?

Given a relative amount of benefit, with a ratio of Watts/Benefit, a lower ratio of Watts/Benefit should cost less to the consumer.

So the consumer would care because the service is lower cost.

I'm truly confused by your argument and @pjlegato's ...

It's more expensive to rent a GPU than to buy it.

This whole comment thread started because GP wrote:

>Remember retail GPUs are much more powerful, much cheaper and much more available than cloud GPUs.

So, still, why would a consumer care about performance per watt?

Because you'll have to include its price when you buy it, but not when you rent it. And how the economics plays out then is not obvious without maths.

I can assure you it's cheaper to buy. Otherwise they wouldn't be leasing it.

(ofc assuming you're not going to use it for like a day)

The retail price of renting the GPU includes, among other things, the cost of the power bill that the cloud service pays (or else they'd lose money.) The more efficient GPU will, all else the same, be cheaper to lease.

Lower wattage allows more GPUs per system. For distributed training, that’s allows 8 GPUs per system which would otherwise be very difficult with retail GPUs.

And GPU-GPU interconnects are important for this type of training so putting as many GPUs as possible close together is necessary.

We use the same graphic buses, such as SXM4 and PCI Express 4.0 x 16, as you would find in your system. Theoretically, there should be no difference in performance or compatibility at all.

If you have conducted your tests on shared instances and have observed such differences, then it's understandable. However, I want to emphasize once again that we provide dedicated virtual machines, which means that the resources are exclusively allocated to each user and not shared.

> Theoretically, there should be no difference in performance or compatibility at all.

At least in the past there was a tendency by NVIDIA to run their professional GPUs at a lower frequency, offering reliability and correctness guarantees as tradeoff. Meanwhile most of the retail/gaming GPUs came overclocked and almost outright guaranteeing that they would start to crash and fail if you tried to run them 24/7.

Even so, an A100 is not actually faster than an RTX 4090, right? So basically the reason you would want to use something like this instead is because you need the VRAM?

Yes, 0x008 makes a valid point regarding this topic. Data center GPUs and RTX Series use different technologies and optimizations for different purposes.

Our intention is not to discourage anyone from purchasing their own GPUs. We offer our GPU cloud services as an alternative for those who may not have the resources or prefer the convenience and flexibility of renting GPUs on-demand. We apologize if the discussion took a different direction, and we appreciate your understanding.

That’s not the case. It is significantly faster, at least for deep learning workloads. Check out Lambda Lab’s benchmarks.

https://lambdalabs.com/gpu-benchmarks for those who are curious, ends up being quite the difference no matter which way you slice it, up to 60% better. Of course, this is probably due to the increased interconnect bandwidth and memory rather than raw compute horsepower, but for the workloads in question that's relevant.

Would you be so kind to post these results? How much faster, which models etc.

I found some charts here showing A100 roughly doubling a 3090: https://bizon-tech.com/blog/best-gpu-for-deep-learning-rtx-3...

With the 4090 being a fair amount more powerful, it might give the a100 a challenge for about 10% of the cost. That’s still about 10k hours of renting one with this service though.

Sure if you are using tiny net(by today's standard) and small batch sizes, A100 is just twice as fast compared to 3090 due to overhead of framework and memory movement. A100 has 312 FP16 TFlops[1], compared to 40 and 82.6 of 3090 Ti and 4090 respectively[2], and has 3x memory. Also it has 10x inter GPU communication bandwidth.

[1]: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent...

[2]: https://images.nvidia.com/aem-dam/Solutions/Data-Center/l4/n...

I believe you are wrong. The relevant metric is tensor flops and 4090 has 330. I believe 312 you quoted for A100 is tensor flops.

Reply on news.ycombinator.com