For those that are interested, I was able to attend a talk by the guy that started Graphcore. His experienced approach to customized chip design for ML/AI is a really great thing to tap into. Some of the themes were: maximize power usage as its a limiting compute factor, remove off-chip memory, and scale horizontally at the individual processor level rather than "sharing resources across cores".