Looking back, the maths taught in elementary school is very basic, too. Just multiplication and division of integers, not even fractions yet.
To be honest, I don't care that much anymore if somebody doesn't want to learn maths. It's just their loss. But maths was one of the few things I enjoyed at school, so I absolutely see no point in removing it from the curriculum.
There are some school forms (like Montessori I think, quite popular here in Germany), where pupils can apparently choose for themselves what they want to learn at any particular moment. Not sure how exactly it works, but might be a good idea.