Personal opinion: Skim milk tastes like a distant relative to whole milk. That 3.25% milkfat makes a major change in the taste of the product. It's like adding butter to a recipe - it's the fat that really adds to the taste.
Taste varies from one individual to the next. I try to avoid skim milk because whole milk simply tastes much better.
Skim milk tastes like whole milk that's been watered down. Simply unpleasant.