I feel very similarly. When I want to understand from scratch, that means no pytorch, no numpy, etc.