It's context. The purpose of memorization, etc. is so that when you look at more complex ideas and concepts, you're not wasting cognitive load on foundational concepts like basic arithmetic.
To directly answer GP's question:
I took a differential equations class last semester, where the easy part of quite a few problems was to solve a quadratic equation. And it was easy, because my algebra classes had correctly required me to practice that skill (and memorize the quadratic formula).
So, you say, I'm never going to take differential equations. And you very well might not. I have two responses to that:
1) That's true, you might not, but the guy sitting next to you will, and he doesn't even know it yet. If you were allowed to opt out of that subject, you both would, and he wouldn't even have a chance to make up for it later.
2) There are plenty of other subjects where the easy part is an algebra concept, and it's only easy because you practiced it years before.