Would you mind defining "overhead"?
Transferring data to-and-from the PCIe / GPU takes time. If the CPU is faster and can perform the calculation before the PCIe is done transferring, then there's no point even touching the GPU.
PCIe is on the scale of ~5000 nanoseconds (20,000 CPU cycles assuming 4GHz). A PCIe write, then read could be ~40,000 CPU cycles or so, and it turns out that a lot of things can be done in that time before the GPU was even _NOTIFIED_ that there was work to do.
CPUs can contact other CPU-cores within 50 to 500 nanoseconds or so, depending on the distance. A 64-core CPU could then have 64-cores * 18000 clocks == ~1-million CPU-clock cycles before the GPU gets any message at all.
Locality of reference ("Data locality") > Spatial and temporal locality usage : https://en.wikipedia.org/wiki/Locality_of_reference
Memory hierarchy https://en.wikipedia.org/wiki/Memory_hierarchy :
> Most modern CPUs are so fast that for most program workloads, the bottleneck is the locality of reference of memory accesses and the efficiency of the caching and memory transfer between different levels of the hierarchy [citation needed]. As a result, the CPU spends much of its time idling, waiting for memory I/O to complete. This is sometimes called the space cost, as a larger memory object is more likely to overflow a small/fast level and require use of a larger/slower level. The resulting load on memory use is known as pressure (respectively register pressure, cache pressure, and (main) memory pressure). Terms for data being missing from a higher level and needing to be fetched from a lower level are, respectively: register spilling (due to register pressure: register to cache), cache miss (cache to main memory), and (hard) page fault (main memory to disk).
Is it that PCIe is necessarily implied by the debuggable pipeline specified by the von Neumann architecture? https://en.wikipedia.org/wiki/Von_Neumann_architecture
Otherwise, computation within RAM avoids interconnect saturation.
"Neuromorphic" computing, stateful RAM with operators mapped to particle interactions:
Memristor > Derivative devices > memtransistor https://en.wikipedia.org/wiki/Memristor#Derivative_devices
Quantum reservoir computing: https://en.wikipedia.org/wiki/Reservoir_computing#Quantum_re...
But these still need a faster and wider (and qubit) bus than PCIe, too: https://en.wikipedia.org/wiki/PCI_Express
> Lloyd also postulates that the Universe can be fully simulated using a quantum computer; however, in the absence of a theory of quantum gravity, such a simulation is not yet possible. "Particles not only collide, they compute."
Quantum on Silicon looks cheaper in today dollars.
Devide the universe in QFT field-equal halves A and B, take energy from A to make B look like A, then add qubit error correction, and tell me if there's enough energy to simulate the actual universe on a universe QC with no instruction pipeline.
Still, that's around 1800 clock cycles for a 3GHz CPU. Which implies there is a floor, below which you want to keep computations local on-CPU versus communicating over PCIe. Whether it be to a remote system via RDMA, or to a PCIe connected accelerator.
[1] https://www.aravision.com/products/infiniband/pdf_mellanox_a...
I came to the 5000ns estimate because I tested some null-kernel a long time ago on a GPU. Basically a "hello world" of GPU kernels, and I tested to see how long it took for the kernel to be called.
I hear that GPUs have sped up a bit since my test a few years ago, but I'd expect it to still be around 5000ns for a "CPU-GPU ping-pong".
See the flame graph here in the overhead section: https://horace.io/brrr_intro.html
I think pytorch has a genius design and makes excellent tradeoffs. But it's not the perfect solution to every problem. Genericity vs specialization: hard to win on all fronts.