These things are always a balance between cost and benefit, and the status quo is just a temporary equilibrium based on oil prices, labour costs, environmental policy, technology, safety standards, business models, etc. and should change as those do.
I try to avoid getting too anchored to incidental aspects of the present. One way is to turn those questions around: what about the people with relatives and friends that already live too far away to make travel feasible? What about all the places that people already can't feasibly get to? What about all of the business which already doesn't get done due to travel practicalities?
There will always be more opportunities to open up by cranking the dial a bit further; but we have to strike a balance somewhere due to the costs. It's a shame that so many opportunities are already blocked by travel costs, but there's no point losing sleep over it. Likewise, if the current costs are too great (e.g. environmentally), then the dial should be turned down a little. Many people would be disadvantaged, but they'd just get added to all those already disadvantaged by the current equilibrium. It's not, in itself, a reason to avoid dialing things down. The things which really matter are: the relative numbers of people affected, whether the budget could be made more equitable or prioritised (e.g. charging frequent flyers more, to keep one-off prices lower), etc.