I've never met a computer where 2D graphics were faster with a GPU. (Try benchmarking and see for yourself. 2D graphics is mostly blitting pixels, where a GPU will only slow things down.)
Lower latency? Yeah, CPU wins easily.
4k resolution 2D composition? Well, GPU is pretty likely to win this one...
CPU has pretty low latency, easily better than what GPU can achieve. But on the other hand GPU has insane bandwidth, high degree of parallelism and hardware ROPs.
On the other hand if you're wanting to make something with the graphical complication of Mario or Mega Man ( purely sprites - no effects - lighting and other things maybe done with alpha blending ) then you might find it makes sense to stick to working with the CPU.
Besides that, you can render so many more sprites if you properly batch them. I could get maybe a few hundred identical particles going around 60 fps with software rendering. With the GPU, you can make particle systems with thousands of sprites (or tens of thousands with a GPU newer than 2012).