The optimal control framework Casadi has had the ability to compute sparse jacobians and hessians for a long time (maybe a decade?), which come up all the time in trajectory optimization.This not only provides massive speed ups in both the differentiation and linear solver time, but also greatly reduces the memory requirements. If this catches on in machine learning, it will be interesting to see if we can finally move past first order optimization methods.