Pytrees

As an example: if you want to implement logistic regression in JAX, you need to optimize the weights. This is easy enough since this can be modeled as a single value, a matrix of weights. If you want to model a 2 layer MLP, now you have to use 2 matrices of weights (at least). You could treat this as two parameters to your function (which makes the derivative more complicated to manage) or you could concatenate the weights and split them up, etc. Annoying, but managable.

When you get to something like a diffusion model you now need to manage parameters for a variety of different, quite complex, models. It really helps if you can keep track of all these parameters in whatever data structure you like, but also trivially just call "grad" with regard to these and get your models derivative with respect to its parameters.

Pytrees make this incredibly simple, and is a major quality of life improvement in automatic differentiation.

It provides similar functionality but is standalone and does not depend on JAX, TF or anything else.

I believe JAX has a utility for this somewhere, can’t quite remember what this is called.

I typically think of JAX as quite restrictive — but I think the reality is that the only real limit on expressivity is that you can’t dynamically allocate inside of unbounded control flow (e.g. creating new allocations inside of a while loop).

https://github.com/patrick-kidger/equinox

(Now on 1.1k stars so it's achieved some popularity!)

This makes model-building elegant (IMO), without any new abstractions to learn. Quite a PyTorch-like experience overall.