Every dollar spent on art classes or chess club is also a dollar not spent on math, but few are suggesting those endeavors aren't worthwhile. Sports are a big target, but they're a big target because they're wildly popular and a huge part of the culture in many U.S. locales. Do you think youth football, soccer or baseball would go away if schools stopped funding them? Public schools are, for better and worse, one of the most democratic institutions in our society, and the activities they offer students reflect that.
When done right, there's a benefit to running youth athletics through schools: Practice time and other scheduling can be balanced against academic requirements, coaches (who are often also teachers) can monitor how performance is being affected on and off the field, etc.
Like anything else, there are tradeoffs and in a nation with tens of thousands of locally-run school districts some places will go way too far in one direction, but it's Utopian to imagine that, say, kids in rural Texas will stop playing football.
(And this is all without mentioning the many positive influences of well-run athletic competition on students: Health, teamwork, the rewards of focused effort...)