Isn't gym class already at the end-point of removing rigor? It's generally not separated into tracks by ability, it's widely regarded as easy filler, and anybody with any serious interest in athletics does their real training outside of class. It's less "Why stop at math?" than "Please stop at gym!"