Setol: A SemiEmpirical of (Deep) Learning

SETOL is a theory of NN layer convergence. It argues that the individual layers of NN converge at different rates, and the 'Ideal' state of convergence can be detected simply by looking at the spectral properties of the layer weight matrices.

SETOL derives the weightwatcher HTSR layer quality metrics (alpha, alpha-hat) from first principles using techniques from statistical mechanics and quantum chemistry.

SETOL also shows that when an NN layer is 'Ideal', it satisfies the so-called Wilson Exact Renormalization Group condition (called the TraceLog condition)

In other words, SETOL provides empirical layer quality metrics that can be used to determine how well a model is trained or fine-tuned. It can help AI engineers get their AI models to the best state possible.

And while this work uses techniques from theoretical physics and chemistry, you don't need know any mathematical physics to read the paper; it is fully self-contained.

Most importantly, all of the experiments are 100% reproducible, and you can test the theory yourself on your own AI models using the open-source weightwatcher tool.