If we have 2000 people, we need 1000 doctors and 1000 secretary and it takes 8 years of training to be the former and 1 year for the later, won't doctors be paid more even without shortage to take into account risks and length of studies?
In some cases, of course, that's because there's a professional association (like the AMA) that limits the number of new workers in that field.
> Working conditions and pay for programmers are incredible and would be a life changing improvement for most people.
Pay is excellent, as you say.
Working conditions ... may or may not be good, depending on your perspective. Half the people I meet don't want to sit in an office all day. Many others refuse to work at Amazon (or even buy from them) or some of the other big companies because of their reputation and what they're doing to the city. "Brogramming" still exists. I know many programmers who get asked to work late without penalty to the company.
> That the profession hasn’t been flooded with talent is a strong indicator that there isn’t enough talent out there.
I know plenty of ex-programmers and people who would make great programmers if they only wanted to. From my perspective, I'd say the talent exists, and the money is incredible -- therefore the working conditions must be pretty severely lacking.
While prospective purchasers tend to describe a situation in which market clearing price is above the price they are willing to pay as a shortage, it's not. A shortage is when nonprice rationing results in unmet demand willing to pay above the actual trading price.
There's also a planning concept of a shortage in the form of supply inadequate to meet some threshold deemed essential, e.g., by society or authorities (which tends to result in economic shortage as price controls or nonprice rationing are imposed). Market price differences alone don't establish the existence of either type of shortage.