cavisne

your data is likely old. IB demand has gone pretty wild since the AI boom.

Only because of Nvidia's margins. "Worse but cheaper" is actually great for a company of Googles scale, but it doesn't make for a particularly compelling press release or paper.

Large clouds tend to be not very good for GPU clusters. The security & management of multi-tenant GPU's is very complex. They have to buy from Nvidia so there is no negotiating leverage. And they know you will be a high maintenance customer (wanting all nodes to work at all times).

So there is no big price or availability advantage for a large cloud (unless you are large enough to rent a dedicated cluster from a large cloud)

ML training is a tightly interconnected HPC style workload (network bound), which is a market Infiniband has been targeting for a long time.

Nvidia saw that and bought Mellanox, and made NCCL/GPU's work really well with Infiniband.

Large public clouds already have huge investments in ethernet and don't want to be further locked into Nvidia, so Nvidia does have a roadmap for ethernet GPU clusters (roughly 1 year behind Infiniband).

But if you are building your own Nvidia cluster, it would be silly to build it on ethernet. Just buy exactly what Nvidia recommends, you are already locked in anyway.

They are focusing in the press release and the report on DC energy needs though. I see that as foreshadowing that things will be worse going forward.

I ran into this in France, where the ticket station used QR codes & NFC. The QR code readers are scratched up meaning printed copies without backlight didn’t work. And iPhone opens Apple wallet when you bring it near a NFC reader, hiding the QR code on your screen.

This awesome talk [1] from OpenAI covers this topic quite a bit, one useful takeaway is how GPU compute is basically static, gone are the days of autoscaling as there is nothing to autoscale to.

I think that beyond optimizing batch size, massive training clusters tend to benefit from scheduled maintenance periods where everything gets fixed vs rolling fixes (as you either need everything to be working or you need to restart the training window). If OpenAI could interleave batch inference with training specific HW downtime like interconnect maintenance it would be another basically free source of GPU FLOPS.

[1] https://www.youtube.com/watch?v=PeKMEXUrlq4

Good video on the issues with extension cords in the US

https://www.youtube.com/watch?v=K_q-xnYRugQ&t=1131s

The Programming Massively Parallel Processors textbook has a chapter on histograms. You have basically mentioned the main trick though (privatization).

For this problem I think coursening would also help (so everything isn’t queued up on the copy to global memory)

How does this compare in bandwidth and latency to nvlink? (I’m aware it’s not available on the consumer cards)

Nothing compared to Nvidia though. The FLOPS and memory bandwidth is simply not there.

Is there an equivalent to this reference for Intel Gaudi?

https://docs.nvidia.com/cuda/parallel-thread-execution/index...

Yeah, I think AMD will really struggle with the cloud providers.

Even Nvidia GPU's are tricky to sandbox, and it sounds like the AMD cards are really easy for the tenant to break (or at least force a restart of the underlying host).

AWS does have a Gaudi instance which is interesting, but overall I don't see why Azure, AWS & Google would deploy AMD or Intel GPU's at scale vs their own chips.

They need some competitor to Nvidia to help negotiate, but if its going to be a painful software support story suited to only a few enterprise customers, why not do it with your own chip?

Very unlikely, Bezos & Gates basically did that and they are still massive targets of criticism.

If Elon had decided to just roll over a couple years ago and do everything the media asked him to do (shutdown the Fremont factory during covid, stop "polluting the sky" with starlink, rely on a nice unionized automaker like GM to "lead the EV revolution") Tesla & SpaceX wouldn't exist today.

They operate on outdated fab's (roughly state of the art - 1)

https://en.wikipedia.org/wiki/Tensor_Processing_Unit#Product...

They also do have a serious presence/spend on things like HBM, semianalysis has some good pieces on this.

Considering private repos used to require a paid subscription misremembering seems likely.

It used to be public by default, and enough people got confused by this that AWS & Github used to specifically scan repo's for accidentally public AWS credentials.

Comes from here [1]. Basically 100 racks of H vs 8 racks of B.

I think there may be a typo though, I assume this also includes liquid-cooled vs air-cooled.

[1] https://nvdam.widen.net/s/xqt56dflgh/nvidia-blackwell-archit...

Could you share the paper? Found this but not sure if it is the same concept

https://arxiv.org/pdf/2307.02666.pdf

They already have.

LLM training is arcane and expensive to experiment with. So OpenAI had to waste a lot of time and GPU-hours on things that didn't work to learn the tricks that did work.

Most of the competitors have lineage straight back to OpenAI, eg the lead of x.ai was previously at OpenAI and Deepmind. Likewise with Mistral and especially Anthropic.

One of the interesting things when weights are open sourced is the community can often improve the results. See all the bugs fixed in Gemma for an example.

Probably https://www.youtube.com/watch?v=nOxKexn3iBo (or just skimming the attached colab).

Causes every thread in the block to wait until they have reached this point. Worth reading a cuda primer for more details on blocks/warps.

Since the threads are relying on each other to fill the SRAM with all needed data if you didn’t wait then values would be missing.

Depending on how they treat oversubscription, Wireless ISP's cant really offer this speed reliably, so they would lose their subsidy.

So building housing in a remote area only served by a WISP gets more expensive.

In practice I don't think the FCC really thinks about oversubscription.

Spanners interleaved tables seem like a similar solution, ie you interleave customer data so it all ends up on the same set of hosts for performance, while still having the ability to create transactions across customers.

I suspect thats basically it (operations one by one and then pipelining to saturate). Thats basically what Groq does also AFAIK. From their website it seems the chips are designed to be connected together into one big topology, the "Galaxy" system. Also similar to TPU's, although they use HBM with only a few very powerful "cores" vs DRAM with low powered cores.

My priors were that Anthropic is the most left leaning & most secretive of the big labs. So this is quite surprising!

A recent presentation on the architecture

https://youtu.be/WQDMKTEgQnY?si=W0E9Kq6P280l3Wcl

IMO we still need an MLPerf submission or similar to really understand if this is more efficient or more efficient only if you also want to minimize latency.

Nvidia has pulled enough rabbits out of the hat when it comes to MLPerf I’m still not convinced they can’t work some CUDA magic and undercut them on efficiency.

Reading this was awesome, but also makes me so glad I bought a Tesla.

What a nightmare vs a simple fixed price.

Doesn’t meta famously not do much testing at all? Ie they use experiments to “test in prod”.