It used to be that to be a composer, first you learned the piano. The piano was thought to give one an appreciation of the nature and nuance of music. From there you could much more easily learn other instruments and then learn how to put them together into something beautiful.
Math is the new piano. Learn symbolic manipulation -- and I'd say that means first-year calculus -- and you can work with the kinds of symbolic systems you'll find everywhere else in the modern world.
I used to do some tutoring when I was a kid, and I think this level of understanding is achievable by 90% of the population given the right environment. The real question is "why are the structures and environments for education that we create so unable to accomplish this?"
Dumbing down the system isn't the answer. It's not like you can dumb down the world to make up for your inability to prepare students for it.