You have to go to the ballpark nearly every single day for almost 8 months. I'm from Detroit and I'm looking at the Tiger's schedule right now. In September they have TWO days in the entire month without a game. By that time they will also be playing doubleheaders or make up games because of weather. In May they go 21 consecutive days with a game. Even if it's not as physically demanding as hockey or american football, that's exhausting. Imagine having to go to the office that much.
On top of that you're playing until nearly November if you're in the World Series and then spring training starts in Feb (it's already started). There's nearly no offseason.