See that's where you're not thinking it through.

Construct the neural nets as lisp list structures. Now you can write functions and macros to operate on the internals.

The self-modifying nature of lisp allows ideas like optimizations, recognizing sections that have rich internal structure but sparse connections as a hint that there is some "clustered idea", etc.

The "it's just matrices" crowd have not really explored the deeper ideas.