What is it that makes higher order derivatives less useful at high dimensionality? Is it related to the Curse of Dimensionality, or maybe something like exploding gradients at higher orders?
In practice, clever optimisation algorithms that use the 2nd derivative won't actually form this matrix.