And that's the issue : climate change requires a global effort and if we want to have the masses to adapt, then we must have the elite to adapt equally.
The key stat is it'd be enough to reach the UK's target on carbon emissions from flying by 2050, at a cost of 0% for the first flight and 9% on the second (going up steeply from there). Of course, other countries may have different data but the principle should be the same.
The vast majority of flights are taken by a small minority of people; they're taken by the richest. Take the United Kingdom as an example: 70% of all flights were taken by 15% of people [0]. It certainly isn't the poorest taking these flights.
So taxing flights will have less direct impact on the poorest than one might think.
Having said that, I think a charge on carbon is more fair than taxing flights directly.
[0]: https://fullfact.org/economy/do-15-people-take-70-flights/
>A carbon fee and dividend or climate income is a system to reduce greenhouse gas emissions and address global warming. The system imposes a carbon tax on the sale of fossil fuels, and then distributes the revenue of this tax over the entire population (equally, on a per-person basis) as a monthly income or regular payment.