“Hiding” network latency for fast memory in data centers
news.engin.umich.edu
news.engin.umich.edu
From a GPUs point of view, this isn't latency hiding, this is latency reduction.
In GPU terminology, using pre-fetching and caching to reduce the latency for a read is called latency reduction. This approach is used heavily in CPUs.
This is in contrast to the main approach used in GPUs, where you just switch threads if you're waiting on a read, dubbed latency hiding. GPUs are able to do this terribly quickly, so there's little worry about thread-switching overhead. It's analogous to 'simultaneous multithreading'/HyperThreading in CPUs.
Latency-hiding is vital in GPUs, which handle massively parallel workloads and use graphics RAM, which has high throughput but poor latency. If you give a GPU a single-threaded workload, you can expect it to stall on reads far more than if you had a CPU handle the equivalent workload.
(Disclaimer: my GPU knowledge is rather stale.)
"Latency reduction", then, would mean "reducing pipeline stalls per execution unit", while "latency hiding" would mean "hiding the fact that some units' pipelines have stalled, by finding other work for their ALUs-and-such to do, while waiting for them to unstall."
Giving a GPU a single threaded workload would be like having a sausage factory start up to make a single sausage, it's inappropriate use of the tool, which will always give poor results.
I see [0] that what I should have said is this (taken from page 7):
High throughput is obtained through the use of wide memory buses and specialized GDDR (graphics double data rate) memories that operate most efficiently when memory access granularities are large. Thus, GPU memory controllers must buffer, reorder, and then coalesce large numbers of memory requests to synthesize large operations that make efficient use of the memory system.
Which roughly translates to higher latency from the point of view of any particular wavefront/warp, compared to the cache-heavy CPU approach. Of course, as the GPU's cores can easily switch between wavefronts, the latency from the wavefront's point of view isn't all that important; it's the overall throughput that counts.
> Giving a GPU a single threaded workload would be like having a sausage factory start up to make a single sausage, it's inappropriate use of the tool, which will always give poor results.
I don't like the analogy, as the single-threaded job could be enormous.
Offloading a very small job onto a GPU is a bad idea because of orchestration overheads. Offloading a large single-threaded job onto a GPU is a bad idea because of poor core utilisation (you're using just one of the lanes of just one of the cores). Offloading a fixed-point-intensive job onto a GPU may be a bad idea because of inappropriate instruction-set. There's the matter of wavefront-coherent flow control. And if the job has chaotic memory access patterns, we're back to our original topic, and you'll get bitten by the wide memory bus.
[0] https://www.cs.cmu.edu/afs/cs.cmu.edu/academic/class/15869-f...
Anything that isn't an embarrassingly parallel workload is a bad match for a GPU. Use tools appropriately.
An embarrassingly parallel workload on 64 bit integers, without wavefront flow-control coherence, and with highly chaotic memory-access patterns, might not fare well on GPU.
I actually like that kind of cycle squeezing, much more than I like web development or other 'high level' plumbing.
Fun problems are always the rarest.
I wonder if now would be a proper time to revisit barrel processors.
Writing programs to take advantage of RDMA, and use of advanced cases like scatter-gather across multiple RDMA servers almost always benefits from building your compute logic around the data access pattern you have.
What if the application had a richer way of communicating with the IO layer instead of a series of reads?
For example reading and processing data in a loop, it could first say that it will need the data at 1a, then x time processing, then 2a, then x time again, then 3a, etc. The IO layer learns what x is, finds the best access pattern given all its requests and upcoming requests. Although this doesn’t work for cases where your next read depended on the last read, it still unlocks many more usage patterns than sequential prefetching does.
Is this just too much extra work for the application developer? What if instead a process similar to JIT was used?
Bottom line is I think incorporating application level knowledge could improve the prefetching