Internal bandwidth is usually the limiting factor for neural networks, not the ALU.