Is this actually capable of propagating the gradient and training more complex layers though?

A lot of these novel AI accelerators run into problems like that because they're not capable of general purpose computing. A good example of that are the boltzman machines on Dwave's stuff. Yeah it can do that but it can only do that because the machine is only capable of doing QUBO.