That's paradoxical seeming isn't it?

But low-pay + labor shortage is seems to be a common thing.

I would speculate it's driven by the production of a very price sensitive commodity. Lots of people choose the cheapest flights possible regardless of other considerations so a significant rise in pilot pay might make a given airline unprofitable since they can't raise prices to compensate.

So airlines feel strong pressure to keep wages down.

And really, you could say a "labor shortage" is what every high volume, low margin operation complains about but the reality is basically "we pay low wages so it's hard to get enough people to work for our low wages". (Obviously, with airline pilots, it's low wage relative to the cost of gaining the skills etc). We can see similar language with H1b visa proponents and even fast food restaurant operators use it periodically.

Probably one factor is the extreme time lag between someone becoming interested in becoming a commercial pilot and actually becoming a commercial pilot (for example, because of the need to get so much experience). It might be similar with doctors, where the required credentials take years to achieve—so if people say "hmmm, I might want to go into medicine, I think I'll do pre-med" they could still be more than a decade away from practicing medicine.

It seems from upthread comments that the institutions to train prospective doctors are somewhat better-developed than the institutions to train prospective commercial pilots. (My understanding is that this used to be primarily military, but now pilots are coming in through a greater range of paths.) If people think they want to be doctors, they can apply to certain schools, get certain loans, participate in an established process to find internships (which their schools will guide them through), and so on. I don't know that there are really institutions to help the pilots this way.

That doesn't mean that supply and demand somehow don't apply or don't influence people's behavior, but a need for massive amounts of training and credentialing (literally years' worth that themselves come at a high financial cost) surely slow down the response.

I think a large portion of the issues surrounding pilot pay are

1. Airline companies are few and far between, it's basically an oligopoly and getting black listed by one company might impact your job prospects in another

2. Pilots invest a lot of time into qualifying for their jobs and the sunk cost fallacy traps them into looking for jobs flying planes.

Also, there might be some strong self-motivated reasons, not wanting airplanes to go without pilots due to altruism, just generally enjoying flying... These aren't IMO strong financial drivers so they seem like weaker points, but they may provide some influence.

Something I haven't seen brought up: if pilots are passionate enough about flying, then that's what they're going to do no matter how little they get paid. Companies don't have an incentive to give a raise to people who are not considering a different job.

Two words: unions and seniority. Here's a good intro:

http://philip.greenspun.com/flying/unions-and-airlines

It's also worth noting that this is largely a US-only phenomenon. It has led to a lot of junior US pilots heading overseas to fly with other carriers, most notably in the Gulf, although that party is slowly starting to wind down.

There is arguably a shortage of pilots with more than 1500 hours and good salaries for them. There is a glut of pilots with less than 1500 hours, and bad salaries for them.

Pilot unions still exist even for these regional carriers. How has collective bargaining worked so well for 737 pilots but not for a regional pilot?

Same way that there's a shortage and low pay for emergency medical services staffing, unfortunately.

Market response of raising pay is always slower to respond if there are other mechanisms to try first.