Isn't gym class already at the end-point of removing rigor?
It's generally not separated into tracks by ability, it's widely regarded as easy filler, and anybody with any serious interest in athletics does their real training outside of class. It's less "Why stop at math?" than "Please stop at gym!"
Gym class, sure, but why is the football team allowed to let some students play more and run farther?
The same reason the math club can still do pentagon-inscribed-in-an-ellipse problems. (But I'm not actually convinced that the math equivalent of after-school weightlifting would be the actual result, rather than just the families that can afford it fleeing to private schools.)
You're no doubt correct. But gym class for me was something like 35 years ago.