The point is that you don't need any software layers at all to code up a basic neural network implementation. A programming language with basic floating-point operations is all you need. The algorithms are not complicated so even x86 Assembly is practical for this purpose if you're already experienced with it. So the "backbone" can simply be your favorite compiler.
> Do you want to code all the OS from the ground up
If it's a "make your OS" course then yes - a simple OS of course. If you want to become experienced in compilers, then writing a compiler from scratch for a simple language is mandatory for people who want to have solid fundamentals. It's not a coincidence that projects like these are common in Computer Science and Software Engineering courses.