Turns out this is more compute intensive than regular backprop. The win (if any) would be in training on future highly distributed architectures where neighbouring parts of the model have accesss to very high bandwidth (to each other) but communicating to further away parts is more expensive