You should give JAX a go.
Whilst I'm here: shameless self-promotion for Equinox and Diffrax:
https://github.com/patrick-kidger/equinox https://github.com/patrick-kidger/diffrax
Which are neural network and differential equation libraries for JAX.
[Obligatory I-am-a-googler-my-opinions-do-not-represent-your-employer...]
I think the delta between JAX and PT on commodity hardware is just a little bit too small to really create much splash on HN.