I can definitely say that there are many people for whom it is a struggle - even those on college courses.
But a hundred years ago we could easily have the same conversation about simply reading and writing - they (the poor, women, or "lower class" ) luke not be taught to read. But it turns out that if you start young enough, and put enough effort in, 99% of everyone can learn
So, perhaps society is not putting enough effort in, or asking enough effort from, our kids for them all to learn calculus.
As the Agile Manifesto says, the work delivered represents the effort input so far. If society wants all children to learn calculus, we need to pay that price.
Edit: I think that we do need a new conversation about education. We have My father left full time education at 14, at 18 most of my peers left school and only 25% of us went to college. Today it's 50% it the total spend has not gone up in line.
I honestly don't know what kind of world the "universal education" advocates of the 1870s were imagining - but I am damn sure it was not people doing college courses on their iPhones, but it is the world they ushered in.
We need to double down on what worked for the 20th Century (and avoid the, y'know wars and genocide and stuff).
Education will lay at the heart of that.