> The GPU often loses even for small neural nets given the large latency

Apple's neural engine shows that you can live in between those two worlds.

As you said, the trouble is the latency, the programming model is still great.