This sounds a lot like what the Muon / Shampoo optimizer do.