Data processing on modern hardware
lemire.me
lemire.me
1. Big GPU memories, sufficiently flexible compute units, breakthroughs in irregular algorithms & tooling => we found end-to-end GPU computing quite viable for most of our data processing. GOAI (link below) is becoming the next big leap here.
2. Getting data on device isn't the issue: Slow communication part is data warehouse -> device (3 GB/s), not cpu->gpu (30-300 GB/s). No significant advantage to CPU here, and for the slight CPU tax here, I expect network->GPU to become cloud-standard.
3. Multi-GPU & GPU memory is insane: The exciting thing for us is (a) all cloud providers have GPUs and are updating them regularly and (b) Nvidia is providing multi-GPU designs with a lot of fast shared memory (ex: DGX2/HGX2). It's a world where for most datasets, there is more GPU RAM than workload data, and 100K+ threads means a node can destroy a dataset in a few steps. For the bigger datasets that require streaming, GPU memory bandwidth & thread count is a monster.
1+3 => We're playing with interactive (<100ms) tasks on dataset sizes that, as interactive tool devs, we regularly get surprised by.
Ex: For some of our high-level interactive visual analytics task that renders in the browser, we'll run a a GPU compute pipeline end-to-end (5+ GPU kernels), where most subtasks take 0-1ms (so many threads!), maybe repeatedly for simulations, and within 20ms, get back to the CPU, and then send down to the browser. Amdahl's Law hits us.. at the network compression & transfer layer.
For a sense of where members of the GOAI (gpu open analytics initiative) community members are going, from the perspective of web devs, we wrote up some planning notes on our path to making regular nodejs apps plug into all this: https://www.graphistry.com/blog/js-gpus-ml-arrow-goai .
In the old days CPUs couldn't saturate DRAM bandwidths but DMA could, so DMA was important. Today a single core of a CPU can read or write about 100 GByte/sec, about 4x more than the bandwidth of a single DRAM module (https://zsmith.co/images/XeonE5-2630v4.png and https://en.wikipedia.org/wiki/DDR4_SDRAM#Modules). Also PCs tend to have more CPU cores than DRAM channels.
If you use something like DPDK to get fast network performance, then I'm not sure what more DMA brings.
If you are already using all your CPU power and not saturating your DRAM channels, then DMA is a win. But instead of buying the fancy NIC with the DMA support, you could have bought more CPU cores.
I can see that the latency and power consumption of DMA might be lower than if the CPU was involved because when a NIC supports DMA, the data path looks like this:
NIC <-> PCIe <-> Northbridge <-> DRAM
If the CPU is involved, it looks like this:
NIC <-> PCIe <-> Northbridge <-> CPU <-> Northbridge <-> DRAM
And you have to jump through hoops to prevent the DPDK NIC polling thread from being descheduled occasionally, particularly if you are running in a cloud VM.
Another interesting thing is that modern machines allow the NIC to write traffic into CPU cache directly, bypassing DRAM (https://www.intel.co.uk/content/www/uk/en/io/data-direct-i-o...). If the data you are trying to read (via DMA) is in CPU cache, then I'm guessing it'll be slower than if you asked the CPU for it.
And another thing: the memory controller in the Northbridge maintains a table of outstanding memory requests that CPU cores or PCIe peripherals have demanded. I don't know what the contention policy is. That could alter things drastically.
DMA is a pretty mature technology these days, it was being used in trading systems a decade ago. Author of this piece is pretty far behind the state of the industry, let alone state-of-the-art. RDMA over Infiniband, yadda yadda.
Can I write portable code that works on all the different vendors' RDMA hardware? Do the major cloud hosts support it without imposing other restrictions? Do I have to re-write my software when I run it on a machine that doesn't support RDMA?
I don't know. I think if you know the answers, you are at the bleeding edge :-)
Noone using this stuff cares about that :-) You would be totally replacing the hardware every 2-3 years anyway (at most) and rewriting everything to take advantage of new hardware features, new CPU instructions, everything. It’s the same in games consoles, you write for the hardware and noone will sacrifice performance for portability that they know they’re never going to see the benefits of. Or in supercomputing.
I think it would be nice if you answered the author's question rather than attack him
Sorry for the snark but this does claim to be a piece on “modern hardware” and it just... isn’t.
I do. I expect the Mellanox ConnectX 3s in the Azure machines I develop on will be obsolete by the time my software is deployed. I don't even know what they will be replaced with, let alone have access to such exotica.
I think my choices today are POSIX sockets or DPDK. I hope for a day when I can try out this new fangled RDMA thing :-)
I do accept your point that these things were possible a long time ago but I think it takes time for them to mature enough for mainstream adoption (ie for use in software you might expect to keep working for decades on a variety of hardware). I think Lemire's question of, "Building software on top of that requires some tooling and it is not clear to me how good it is right now", remains open.
It's fairly straightforward, you use MPI, or code it yourself (so basically, you use MPI).
https://wiki.mpich.org/mpich/index.php/MPI-3_RMA_Implementat...
https://www.open-mpi.org/faq/?category=openfabrics#roce-supp...
maybe you use PVM, I think there are RDMA extenstions. But really, you probably use MPI.
I'll hopefully get chance to do some experiments soon.
I thought you made good observations, I was trying to add to it. Cheers.
At least for x64 systems, I'm almost sure this is wrong. I think your linked image on zsmith.co must be for multiple cores, and probably for multiple sockets.
On all other tests I've seen and run, single threaded memory bandwidth is much less than full system bandwidth. Here's a fully labeled chart showing results more like I'd expect: https://www.anandtech.com/show/11544/intel-skylake-ep-vs-amd...
In this chart, a single core of an Intel "Broadwell-EP" Xeon E5-2699v4 (bigger sibling to the core you link) only gets 18GB/s, while 8 cores on the same socket max out at 60GB/s. Only for the whole machine (with two sockets and thus two memory controllers) are throughputs higher than 100GB/s seen.
In practice, the limiting factor is the number of outstanding read requests supported per core. In accordance with Little's Law, you need about 100 requests in flight to overcome the latency of a read from RAM. On Intel, each core can only keep track of 10 outstanding requests. Prefetch can effectively double this, but you still need 4-6 cores to saturate a modern memory bus.
See here for more details: http://sites.utexas.edu/jdm4372/2016/11/22/sc16-invited-talk...
If instead it's my understanding that's wrong here, I would love to be corrected!
Aren't databases well understood in comparison to machine learning? I know it would take me a lot less time to look up how to tune a database than it would to get a machine learning system up and running.
Of course, the real optimization should be for legibility and maintainability... human time being far more important than machine time. But that wouldn't allow fetish discussions diving in to low level efficiencies, now would it?
Any sufficiently complex system acts as a black box when it becomes easier to experiment with than to understand. Hence, black-box optimization has become increasingly important as systems become more complex. - Google Vizier: a service for black-box optimization Golovin et al., KDD'17
There's a point up to which this would be better handled with a rules system. And after that "clever" stuff can search for rules. In any case the real problem with using ML is getting enough data. Most folks don't feel like spending millions of dollars waiting for their database autotuner to climb billions of hills, they'd rather someone else did it for them. Which brings us back to prebaked rules systems.
[1]: https://www.youtube.com/playlist?list=PLSE8ODhjZXjYplQRUlrgQ...
Why not? People use CPUs all the time where GPUs are much more efficient, why not the other way round as well? If you have GPU, why to keep it idle?