> GPUs have hundreds to thousands of cores

That's not exactly true. The GPU equivalent of a CPU core is called a Streaming Multiprocessor (in NVIDIA language), and it's latest GPU, GTX 2080 has 72 of them.

Each of these SMs can run hundreds of threads, but they run the same code in lock-step.