Algebraic Simplification Neural Nets
Trivially a node that has a zero weight can be removed as can any links to / from that node.
It should also be possible to eliminate nodes that have a full-value (aka '1' on the 0-1 scale)
I have also seen work where the matrix multiplies during training can have columns "collapsed".
The ultimate question might be applying an "algebraic simplification" of the final network to simplify a post-trained network used for inference.
The idea is to take a path through a network, constructing the equation for that path, reducing it to a shorter path by conbining nodes and weights.
It is certain that a node participates in several (hundred?) paths. In this case it might be useful to "copy" the node so it can be part of a path reduction without affecting other paths.
I believe that in theory some neural networks can be reduced to a single hidden layer[1]. The game would be to algebraically reduce network depth.
[1] Lee, et al. "On the ability of neural nets to express distributions" https://arxiv.org/pdf/1702.07028.pdf (2021)