To some extent you're right, the benefit of a higher-level abstraction is using it without knowing the details. For standard usage, on the "happy path", this is fine. But if you need to modify techniques, or debug them, it's kind of impossibly frustrating without actually knowing what you're doing!
BTW computers have much cleaner abstractions than mathematics. e.g. the JLS defines Java independently of hardware; IEEE 754 is similar for fp arithmetic. There are specifications all over the place.
But my experience with mathematics is completely different - you have to understand the lower level to understand the next level.
In my personal journey, I started off with your perspective of just learning the higher levels that I directly needed. It was very difficult, but after heroic efforts, I made breakthroughs! After a while, I noticed these "breakthroughs" were mostly entirely to do with material from lower levels... So I went back to them. This happened again and again, going lower and lower. Now I'm basically re-doing high school maths.