The original paper [1] used LBFGS [2], it is quasi-second-order optimization algorithm.

  [1] https://arxiv.org/pdf/2404.19756 - "Both MLPs and KANs are trained with LBFGS for 1800 steps in total."
  [2] https://en.wikipedia.org/wiki/Limited-memory_BFGS

(Quasi-)Newton methods approximate learning rate using local curvature which gradient-based methods do not do.

The post relies on Tinygrad because it is familiar to author and author tinkers with batch size and learning rate, but not with optimizer itself.

I think that even line search for minimum on the direction of the batch gradient can provide most of the benefits of LBFGS. It is easy to implement.