Comment by fluffything | Hacker News Reader

Comment by fluffything | Hacker News Reader

Parent Full thread

It's not a dumb question, that's just a problem that most people don't have.

If you have hundreds of 80.000$ nodes with 8 V100 each consuming thousands of Watts per node, you start thinking about how much money is the 90% idle time of those nodes, waiting for data from the network, is costing you (probably millions of dollars per year).

So yeah, a network that's 10x faster is worth every penny if it can turn the system from a 90% idle to <10% idle.

That's why people always pay extra for NVLink, HDR Infiniband or HPC Ethernet interconnects, and also why Nvidia bought Mellanox last month...

Essentially, when your machine has 1 ExaFLOP/s of compute performance, every second that it remains idle waiting for the network you... well lose 1 ExaFLOP. That's just a lot.

Also, imagine millions of cores writing to the same harddrive simultaneously - hell try doing `ls` on a normal linux machine on a folder with 10 million empty files. The "storage" infrastructure to support instantaneous I/O on these systems is quite complex as well. Every second that the system is writing to disk you are wasting another ExaFLOP of compute.

I was going to say it's hard to make a single server that costs $80K, but I checked with my preferred white box vendor.

Two Xeon 8280 CPUs (28 cores each), 10 nVidia Tesla V100 32GB cards 1536GB DDR4-2933 memory, registered ECC, 24 DIMMs 4x 3.8TB SSD, SATA, Intel D3-S4610 Series and a Mellanox ConnectX-5 IB InfiniBand, dual port EDR 100Gb/s

is ~$160K.

That is indeed some very expensive idle time (although, I think I'd work as hard as possible to get my problem to fit on individual machines with data on external fast storage).

Yeah.

FWIW I think it would be rare for somebody to build that kind of node. Applications that fully utilize two CPU sockets while simultaneously maxing 10x V100s are super rare. Usually for GPGPU applications the CPU is mostly idle, and only "orchestrates" the GPU usage.

If one needs to support both GPGPU-only and CPU-only applications in a cluster, one would usually just build two separate partitions. One with e.g. 4-8 V100s per node, but way weaker and cheaper CPUs, and the other one with 2 beefy CPU sockets but no GPUs.

Also, if you are going for that kind of interconnects, you are probably building a system with >100 nodes. For your price, that's 160 million dollars, which is on the ballpark of what these systems costs. You usually get a cheaper price per node at that scale (although not much cheaper), but there are many costs (installation, maintenance, training, etc.) that are not covered there. In particular, those nodes will be running quite hot (just add the watts that those V100s and the CPUs use!), so the cooling infrastructure required to held those in a relatively small space can also end up costing quite a bit. You don't want a water pipe breaking and destroying a rack containing 10 of these nodes costing 1.6 million dollars, right?

On many-GPU machines, the CPU isn't idle- it's handling the data infeed and data outfeed pipelines. This can often be a bottleneck if training is really fast and you need to stream lots of examples/sec.

As for your comments on water pipes breaking: I work on Google's TPUv3, which uses water cooling to the TPU chips, see https://www.nextplatform.com/2018/05/10/tearing-apart-google... for some speculation on the details.

> On many-GPU machines, the CPU isn't idle- it's handling the data infeed and data outfeed pipelines. This can often be a bottleneck if training is really fast and you need to stream lots of examples/sec.

This depends on both the application and the actual machine. Moving data from storage or from the network to the CPU to then feed it to the GPU just increases latency. Unless you are doing some non-parallelizable pre-processing of the data in the CPU, you are often better off with doing DMA or RDMA directly to the GPU memory, fully bypassing the CPU.

Not all machines support that, and not all applications support that either, but many applications do. Arguably, even for the applications that do this, leaving the CPUs unused is not really a desirable property. I've seen a couple of applications that split the compute workload to the CPU to fully utilize the CPU as well, and I've also seem other applications that once the GPU finishes processing they move the data to the CPU memory to leave space on the GPU for the next batch as quickly as possible, and the CPU then streams the data off the node while the GPU continues to compute (so you get an RDMA->GPU->CPU->Storage/Network loop).

> see https://www.nextplatform.com/2018/05/10/tearing-apart-google.... for some speculation on the details.

Thanks for the link, I never learned much about this part of the system.

But can't you make the same argument about bandwidth? Every second that the processing units are computing, terabits of bandwidth are unused.

Except the goal is usually to compute, not to transfer data, so the interconnects are a necessary evil.

Ultimately what everybody wants is a balanced system (that acts balanced on the workloads of interest). Put another way: you don't want any one part of the system to be massively overprovisioned in a way that causes expensive bottlenecks. The Cray T3E had fairly slow processors, but you coudl keep the entire supercomputer at very high utilization and get better throughput that machines with faster processors and a slower interconnect.

Good question. Of course you can make the same argument about bandwidth, the question is whether you want to make such an argument.

If you are building a system whose purpose is to transfer data (e.g. a router), every second that system is "computing" and not "transfering" you are paying for Watts that aren't doing anything useful.

A supercomputer purpose is to compute stuff, not to move data around, so making that argument about a supercomputer doesn't make much sense because nobody cares whether the supercomputer is using all its bandwidth or not as long as its using all of its compute capabilities.

So that's like making the argument that everything that isn't a plane, well, isn't a plane. Sure, my guitar isn't a plane, but it is not the purpose of my guitar to be one :D

No, computing is happening in parallel with data being sent/received

What fraction of communication on a typical HPC system is overlapped with computation? My guess is rather little, despite the Mellanox sales pitch. I do wish people commonly made cluster-global profiling-type measurements to be able to answer that sort of question and do optimization.

Heavily depends on the application.

Many applications running on a Top 10 system of the Top 500 are required to show good single threaded performance and good weak and strong scalings up to a big part of the system.

Checking all these 3 check boxes is hard to do if your communication is not fully overlapped by computation. The smaller your problem, the better your single threaded performance (e.g. at some point your problem fits in L3), and the larger fraction of execution time that communication dominates.

Even the poorest applications on these systems use non-blocking MPI to overlap communication with computation nowadays.

Reply on news.ycombinator.com