Neat, I love projects like these.

The next level down is to do it directly in numpy.

And then from there, write a minimal numpy work-a-like to support the model above.

You start with a working system using the most powerful abstractions. Then you iteratively remove abstractions, lowering your solution, then when you get low enough but still riding on an external abstraction, you rewrite that, but ONLY to support the layers above you.

Following the above pattern, you can bootstrap yourself to have full system understanding. This is not unlike RL+distillation that human persons do learn complex topics.