GPUs are latency hiding engines...they address the mismatch between processor clock speeds and memory latency by a unique scheme. Since they can’t improve memory latency, instead they parallelize memory and vastly increase the bandwidth available.

Once they have increased memory system bandwidth to be able to feed the multiprocessor throughput, the rest of the architecture is designed to make the most efficient use of it.

They spawn thousands of threads and schedule them in and out really quickly... so the processor utilization is always as close to 100% as possible. When a thread is waiting for memory it is put to sleep in a few clock cycles. When it’s data is available it wakes up and does it’s business. Since the workload is split among thousands of threads therefore there is somebody ready to be scheduled. This is why GPUs only make sense to run on massive workloads.

The same trick can be used to hide SSD latency.