What you're talking about here is the client/inference side, not the server/training side of the ML/DL compute battle. The former is meaningful in a sense, but practically insignificant in relation to the latter. Almost all ML/DL compute GPU, CPU, or otherwise is on the server side of the equation, and if you're relying on vector CPU math for it, well good luck.

Accelerators have a very clear space for themselves on the training side, and no amount of fancy ISA magic is really going to make up for the fact that TPUs and the like beckon. It's where most of the real money is.