Anecdotally, PE in US wasn't worse, compared to Japan. (Where I have been for first 6.75 of school.)
Grading in Japan was fairly black and white; whether you can do certain thing or not. It sucked even worse being thrown into some team sports with no briefing on rules.
At PE classes in US, where I spent most of middle and all the high school, I think the problem was more of how compatible I was with the teacher. Some graded very strictly on the outcome (which sucked), but some other graded on efforts (which was more reasonable) -- some had good balance while others were worse.
Problem of physical fitness is that there are quite a bit of genetic that plays huge role. You can only improve your athletic ability so much to meet some standard. Thus, unless you are training for the Olympics, it really should be graded based on how much you improve over time rather than whether you can reach certain goal that may be unrealistic to some.