Strictly speaking, GPU's have thousands of ALU's/execution units, spread over a few dozen full cores at most. And reaching full utilization of that amount of compute is not always easy either, many problems just don't parallelize that widely.