Right. It's not "hard" in the grand scheme of things of climbing 8000m peaks in the Himalayas, highly technical climbs, or even a peak like Denali. However, it's also a far way from pull up to a parking lot and get a little exercise for the vast majority of people. Altitude makes a big difference for most. As you say, there's acclimatization. And also there's a world of difference climbing at 5000m+ relative to climbing relatively near sea level for almost everyone.