And yet, GPUs have hundreds to thousands of cores, and we find them very useful...

There are many, many useful tasks which are embarassingly parallel or nearly so. For those tasks, doubling the core count doubles the performance. Single core gains have stagnated, so a performance doubling is a huge win. There's no other way to just up and double performance for any class of problem.

And even outside those tasks, more cores means more simultaneous heterogeneous workload, which means deeper and richer information pipelines. If you're live editing audio or video, for example, your core count determines the number of plugins/tracks you can work with simultaneously - and it's not unusual to have hundreds of tracks and dozens of layered plugins.