What if you abandoned gradients/back-prop entirely? They knew that for reinforcement learning this was good clear back in 2017!

https://openai.com/blog/evolution-strategies/

Put engineering effort into creating fast, GPU powered (tensorflow/pytorch) algorithms for neuroevolution of the weights of a neural network. I'm still not convinced that properly leveraged gradient free/Feed-forward only optimizers have been tried yet by researchers - namely because they never actually wrote fast gradient free optimizers!