No, it goes beyond that. There's "arithmetic," the applied usage of addition, multiplication, subtraction, and division to permute numbers, and then there's Arithmetic, the set of theorems and axioms that give rise to that system of applied arithmetic. Memorization only works for the applied part, and children aren't usually taught that there is a system of reasoning behind those rules. Without that, no amount of mathematical dexterity in pushing symbols across a page will help them understand anything past the 100 level, and sometimes not even that.
I also think there's a huge undercurrent of resistance from adults to having children learn that system of reasoning because adults don't understand why it's useful, and in my experience when people don't understand something they dismiss it.
Edit: A nice example of another axiomatic system that might be more approachable is Euclid's Elements, in which five postulates are used to develop a system of geometry using an unmarked straightedge and a collapsible compass that you could, if you were careful, use to build bridges and other large buildings.