My understanding is that GPUs have moved toward what I described: thousands of discrete cores with large amounts of math performance but generally terrible (or even non-existent) branching...
Once they have increased memory system bandwidth to be able to feed the multiprocessor throughput, the rest of the architecture is designed to make the most efficient use of it.
They spawn thousands of threads and schedule them in and out really quickly... so the processor utilization is always as close to 100% as possible. When a thread is waiting for memory it is put to sleep in a few clock cycles. When it’s data is available it wakes up and does it’s business. Since the workload is split among thousands of threads therefore there is somebody ready to be scheduled. This is why GPUs only make sense to run on massive workloads.
The same trick can be used to hide SSD latency.