Programming those would require a new paradigm possibly.
GPU cores run the same operations across a large number of "threads" (32, 64, etc) - they're no really threads.
When you have branching code running on a GPU, the threads are split into the branches, and each branch gets executed separately.
CPU threads run independently of each other (there's often locking for access of shared data, but that's not what I'm talking about).
Even when multiple threads on a single CPU core are running (SMT) they're still not performing the same operation.
Per thread (it's not quite what they are, but a more appropriate terminology escapes me) there is and (unless it's a very stripped down CPU) will always be a far greater silicon overhead for CPUs compared with GPUs.
That wasn't the best description of the way things work, but I'm rather tired and not a hardware guy.
I'm not an expert on GPU hardware but in my readings on this, I've seen it said that if there is a branch in your code, all cores take that branch even if it is a no-op for many of them, thus all branches are taken by all cores when the code is not coherent. Hence why making GPU parallelization can be quite a different programming paradigm to really take advantage of how it works, and it is more challenging to do properly than truly independent CPU threads that do not affect each other when they branch.
Ideally all "threads" of the wavefront take the same side of a branch, so it can skip the not-taken side of the branch. Wavefront divergence is when a single wavefront has threads that take different sides of a branch, so the whole wavefront runs both sides and masks out the results based on branch direction per thread.
Also, sorry about the barely prompted and completely unjustified wall of rambling text - this is something that I really should've given more consideration to already.
That's a programming model that has the potential to go very wrong, very fast if you don't think in depth about what you're doing. Branches with branches are going to be very problematic (with exponentially decaying throughput), although multiple branches at the same level are handled very cleanly.
To be honest, I don't know how I thought that it might be rescheduling things instead.
The sort of rescheduling that I seem to have been thinking of, could only make a difference in cases where the batch size exceeds the wave size, and my guess is that the factor of difference would need to be large to make an impact.
At the base case, NOPs and rescheduling would perform identical operations - so all the scheduler would get you is a hardware overhead (the time overhead could be mitigated when running on a single wave).
The scheduler would also introduce latency since different waves in the same batch would need to wait for each other before rescheduling could proceed.
You'd cause problems for your memory layout - programmers referencing values from threads that have branched would probably need to be treated as undefined behaviour.
You'd also need to introduce a stack, per thread, to keep track of the wave histories - allowing you to unsort and re-reschedule at the ends of branches.
All this to run parallel execution on what seems to be uncommonly large batch sizes (I believe AMD just dropped their wave size down from 64 to 32 - I don't know if this is because 32 is a Goldilocks batch size, or if it's simply to achieve better performance on Nvidia optimised applications).
Perhaps this should be threads, but certainly not cores - it's a single core running the same operations on different data across multiple threadish things.
I'm not sure if there's a more technically accurate terminology for quite what they are.
If you're interested in an earlier mass market phase of moving GP computation of the von Neumann CPU approach, check out the PlayStation 3's "cell" architecture. Devs really struggled with it and Sony went back to von Neumann for the PS4.
CPU cores are spending a lot of transistors minimizing latency: branch prediction, microops fusion, sophisticated multi-level caches. GPUs are fine without most of that (they do have caches but much simpler ones), they are spending majority of their transistors and electricity on ALUs. Instead of fighting latency, GPU cores embrace it and optimize for bandwidth on massively parallel workloads: they have cheap hardware threads so they switch threads instead of stalling the pipeline.
The programming paradigm to enable this exists, it's just pure functional programming. But people are intimidated by Haskell.
We need more investment in functional programming before the dream of a 1000 core computer can be realized.
Whether more cores can be used is ultimately down to the problem, not any language. Some problems are naturally paralisable, some just aren't.
FP maybe exposes a bit more parallelism, but it may introduce more overheads such as less efficient cache use. FP is not a solution, it may be part of the solution.
Parallel programming in Haskell is hard. GHC's style of by-demand lazy evaluation and the ubiquitous use of monads impose a lot of sequential execution.
There has been good research around parallel FP programming languages, but that was mostly around the 90s (Sisal, pH)