You might ultimately only need some specialized "neural processing instruction set" for either the GPU cores or for the CPU cores. Or at least, I don't see any obvious obstacles to that.