The Myth of the Science and Engineering Shortage
theatlantic.com
theatlantic.com
[1]: http://www.c-span.org/video/?317093-1/house-subcmte-hearing-...
I think there's a bit of a rhetorical issue with this, though. The dominant narrative is that we have too many people doing soft, fuzzy, weird, useless liberal-arts degrees. Studying things like literature, history, philosophy, political science. What we need is more people doing hard, rigorous, mathematical, technical STEM degrees. Studying things like physics, biology, mathematics, computer science, chemistry.
If you admit that there is no shortage of mathematicians, though, you undermine the whole strategy of "we need more STEM, less liberal arts", because mathematicians are the rhetorical core of STEM: rigorous, technical, mathematical, non-fuzzy. If it turns out mathematicians are about as useful as historians (useful in principle, not directly in a major applied shortage), the whole narrative fails.
To a certain extent I think the whole STEM construction is based fundamentally on trying to hand-wave across this gap: math and physics are prestigious and perceived as hard/rigorous, while computer programming is in demand. The union (not intersection) of these two fields is STEM, which perceives itself as rigorous + hard + in-demand.
[0] was linked in the article, detailing the unemployment rates and earnings among most college majors, mentioning 7.8% unemployment among recent college CS graduates and 5.6% for experienced ones. This is used to support the claims of the article.
If CS was broken down into specialised fields, like the rest of the STEM, you would get a more accurate picture.
Qualified US workers have this annoying tendency to be unwilling to uproot their life and move to where the jobs are. Foreign workers coming from other countries don't have this issue...they're moving anyways, so they move where the jobs are. And hi-tech jobs cluster in certain areas, usually around one or more universities to supply talent. So you end up with areas that have lots of jobs and not enough qualified applicants and areas with qualified applicants and no jobs.
That's what makes this issue so obnoxious...both sides are right. And that makes it difficult to find a middle ground.
Is there actually a shortage of mobile qualified workers in the US? If so, what is your evidence?
[0]: http://www.bls.gov/opub/mlr/2011/05/art1full.pdf page 9 [1]: http://www.bloomberg.com/visual-data/best-and-worst/highest-...
Consider [0] says that petroleum engineers are one of the highest paid STEM jobs, and [1] concurs. But a petroleum engineer in, say, North Dakota where there's an oil boom is likely going to have a higher salary than one in Hawaii even after adjusting for local wages. There are so few petroleum engineering jobs in Hawaii because it has no oil resources. If 0.1% of oil engineers are are willing to take a pay cut to move to Hawaii, then that small subpopulation may be enough to supply the Hawaii jobs market. (1 out of every 1000 jobs in ND is a petroleum engineer. We know that wasn't supplied by the indigenous population. While BLS doesn't have an entry for petroleum engineers in Hawaii, indicating that it's small.)
In a related vein, [1] points out the nuclear engineering is the highest paying job in Tennessee. This is probably influenced by Ridge National Labs and the three reactors in the state. The University of Tennessee offers a MS in nuclear engineering, so there's definitely a local production of people with those skills.
While Montana (no reactors, no nuclear research) doesn't have a BLS listing for nuclear engineers, which indicate that the demand for such is quite low. Low enough that secondary influences, like the desire to be near good fly fishing, or to move back to family in Montana, may be a bigger factor than cost-of-living.
Thus, regional salary variations, even when adjusted for CoL, may not give the estimate you're looking for.
http://www.washingtonpost.com/blogs/wonkblog/wp/2013/05/11/w...
Uprooting one's life becomes more undesirable.
That would suggest that people are willing to move if there were stronger economic reasons, but the economic disparity across the US isn't high enough.
While the OP suggests that people aren't willing to move despite there being an economic disparity.
Thus, doesn't your link support the thesis the STEM shortage is a myth?
ADDENDUM: downvoters can take a warm piss on a power line.
John Adams US diplomat & politician (1735 - 1826)
FTA: "Because labor markets in science and engineering differ greatly across fields, industries, and time periods, it is easy to cherry-pick specific specialties that really are in short supply, at least in specific years and locations. But generalizing from these cases to the whole of U.S. science and engineering is perilous."
But, surprise surprise, still higher unemployment than the guild-protected professions.
FTA: "unemployment among scientists and engineers is higher than in other professions such as physicians, dentists, lawyers, and registered nurses"
On a related note, a good read about guilds: http://nplusonemag.com/death-by-degrees
"It is true that high-skilled professional occupations almost always experience unemployment rates far lower than those for the rest of the U.S. workforce, but unemployment among scientists and engineers is higher than in other professions such as physicians, dentists, lawyers, and registered nurses, and surprisingly high unemployment rates prevail for recent graduates even in fields with alleged serious “shortages” such as engineering (7.0 percent), computer science (7.8 percent) and information systems (11.7 percent)."
Which is counter-intuitive to me, considering how heavily regulated those other professions are (and their numbers restricted).
There are 13,605 taxi medallions in new york city. What do you think the employment rate among taxi drivers in NYC is?
I'm in a second tier city with more limited options. I know folks who successfully start CS grads in the $50-60k range, mostly from state schools. Someone with a good track record is anywhere from $80-105. Sounds awful, but a 2000 ft^2 house can be found around $200k in a nice area. McMansions are $400-500k.
I ended up moving back here from NYC, taking a big salary haircut in the process. My overall standard of living got a little better right off the bat, mostly due to reduced housing and commuting expenses.
People just don't seem to understand how little money $150K is relative to what a skilled and competent worker could get out of a remotely comparable sum 50 years ago. OK, you make $150K, can your four or five kids walk down the street to a quite good school? Is it essentially optional whether your wife works or not?
The husband was a carpenter and the wife was a bookkeeper.
They had a boat probably worth $10,000 in 1969 dollars and a further $5,000 in cash saved up. I looked that up in the inflation calculator and found out that $5,000 in 1969 dollars is about $30,000 today.
Can you imagine a carpenter and a bookkeeper building up assets of $90k dollars in LA today through a few years fastidiousness and hard work?
No. This is a gross exaggeration and extremely out of touch with economic reality.
I would wager the couple making $300k in SF are not living like that. Hell, I used to live a lifestyle much better than my grandparents did back when I lived in the Bay Area, on a lot less than a six-figure salary. I didn't have kids, but I also didn't even make $50k, and yet I still ate out multiple times per week, not per year.
I know nobody making over $80k in the SF Bay Area who I would not be comfortable calling at least upper-middle-class. The people who think otherwise mostly live in an insane bubble in which they only know people in the top 10% of the income distribution, and have absolutely no idea what a middle-class American lives like.
> sq-ft
Well this sort of thing is the crux of the matter. If you give a shit about restaurants and ipads, well yes now is better. If the idea of having a handful of grandkids appeals to you, well now is worse.
The result is a society in which people find it quite easy to obtain what their parents would have called frivolous luxuries (sushi), but quite difficult to obtain what their parents would have called basics (an affordable house in a good neighborhood).
How about Silicon Valley? Cupertino is safe and has good schools. And it's <1 hr drive to Santa Cruz or Half Moon Bay if you want beaches. Or you could do the other direction, and live in SC and commute to Cupertino (or wherever). Google even runs shuttles if that's your thing. I lived in Santa Cruz for years on a 5-figure salary, and felt quite comfortably well off. Actually I sort of wish it were further from Silicon Valley, because the only bad thing about living in SC was that the rich techies with 6-figure salaries were driving up prices...
Another important distinction to make is that there are almost no science (i.e. research) jobs any more. Most technology jobs are just run of the mill programming, not actually investing in research.
Not everyone has the same experience as you. Furthermore, this article is talking about the entire STEM domain.
If you're reading hacker news, you're already probably doing better on that front than I was during my entire undergrad. I'd highly recommend getting an internship and building a personal project before you graduate too, as I believe my failure to do both of those things (in addition to a sub-3.0 GPA) were the factors that have prevented me from getting a job thus far.
So it will be at least five years before you start to break even. Minus the housing and living costs over that five years.
You spent over $150k and four years ($600k) worth of your time training for this job.
Also remember the market is hot. In 2000/2001 air went out of the bubble. Also in 2008/2009. Right now is the peak of a go-go time like 1998/1999 was. It is not the norm.
Except that it doesn't work, and it should be obvious why. There's no reason to bust your ass in college to enter a field the government floods with foreigners. The more people the government imports the less attractive the field looks to domestic students choosing a major.
I can't complain about my treatment as an EE and later a software developer, but knowing what I know now I doubt I would go into anything technical if I were graduating from high school today.
If I couldn't get in I'd try to get an ROTC scholarship for as prestigious a place as I could manage and graduate without borrowing money. Either business or liberal arts, though the latter case is a minefield of worthless programs, so it would require some research.
That needs to stop but having the best congress money can buy pretty much assures that it won't.
1 - virtually no supposed 5x or 10x programmer makes even 2x more. If people are really 5x, why on earth don't I see $300-$500k salaries? Particularly given the fall in communication costs ala Brooks, it would be an enormous win for employers
2 - at least in the bay area, there would be tons and tons of highly qualified candidates if moving to the bay area wasn't a financial nightmare (cost of living is horrific and the pay doesn't come anywhere close to making up for it); and if buying a decent 3 bedroom home and having kids didn't nearly require winning a startup lottery -- or at least enough for a good downpayment to get to a conforming mortgage.
3 - I'm in my 30s, and in my cohort of developers I know a number (all of them very skilled and in high demand in the bay area) who have moved to the midwest or austin because financially they're so much better off
4 - the majority of complaints about unavailability of developers, particularly in the bay area, are after the fact justifications to (1) cover employers not paying salaries commensurate with the cost of living, and (2) the ability to import (cheaper, more easily controlled) h1b labor
4a - with a side of companies ducking their responsibilities to america, the state they are in, and their communities to help create the employees they need. Now obviously I don't think 1-4 person startups have any such responsibilities, but somewhere between that and google/fb/hp companies have responsibilities to their communities and countries that bay area companies in particular almost completely duck. For example, why isn't facebook or google, in lieu of whinging about difficulty hiring (while illegally restraining wages, ain't that beautiful) running hacker schools themselves? It couldn't be because they'd rather let someone else pay for it and cherry pick the winners (saving money two ways)?
Some reasons you don't see higher salaries:
1. It's very hard to measure the difference in programming ability directly. Good programmers also tend not to realise just how much better they are, and are generally unwilling/unable to demand higher salaries (not because they're programmers, but because most people by default aren't good at these things.)
2. Good programmers tend to cluster around good companies, one of their advantages being that they're surrounded by people at their level. Think Google, Facebook. Over there, if everyone is as good as you but making the same as you, then you don't feel there is a disparity.
3. Some programmers DO make 300-500k. You just don't hear about it.
4. More importantly, some people make 300-500k in roundabout ways. E.g. some great programmers work as freelancers and make that amount, because that's one way to solve the problem of companies being unwilling to pay so much more.
5. Some programmers with more business-fu start consultancies and startups, making lots of money that way in a non-obvious way.
If there was really a shortage you'd see salaries for programmers going up. Instead there are documented cases of companies colluding to keep pay down. Based on the high salaries you could argue that America has a CEO shortage and needs to increase visas for foreign CEOs.
Because it's much easier to measure performance of CEO's, at least artificially. If Google does well, then you (supposedly) know that Larry Page is doing well. That's how the market treats it, at least. But can you really tell me that you're able to tell me which of the 10's of thousands of engineers that Google has is responsible for that success?
Also, there's the issue of leverage. Just like a programmer can build software that is used by millions, and therefore has a lot more leverage and creates more wealth, so a CEO usually influences even more customers than the programmer. Paul Graham has an article about exactly that - the idea of a startup is to give e.g. programmers a lot more leverage than they usually have.
And arguably, the fact that a few engineers can build WhatsApp and earn billions is an example of that in action - getting more leverage by building a company, proving success, then gaining the money it gives. But the average exceptional programmer, while being 10X better at programming, won't necessarily also be better at business or money generation, etc.
Lastly, the fact that there are cases of companies colluding to keep pay down proves the opposite of your point - that salaries should have been even higher!
Originally I wrote a much longer reply but after re-reading it it came across as much more anti CEO compensation than I intended. I don't view short term market movements as an accurate reflection of company or CEO performance. If you willing to use easy to calculate but broken metrics I can supply a few for programmers. How about lines of code? This also ignores under performing executives getting big salaries and severance packages. No one is going to pay me millions if I get fired.
"And arguably, the fact that a few engineers can build WhatsApp and earn billions is an example of that in action"
I view employment at a startup like buying a lottery ticket. Someone is going to get a large payday but that doesn't mean they're better than someone that doesn't get one. Even with the possibility of a big payout your expected return is pretty low.
I think I stated the collusion argument badly so let me try again. If you're in charge of a major corporation and experiencing a talent shortage would you tell your HR department to exclude a large portion of the local candidate pool? That's what those companies did.
What is clear is that, assuming CEO's do affect the value of a company significantly, they'll tend to affect it much more than even the best programmers, especially at a company the size of Google. That's just an issue of leverage. So a (possibly semi-broken) method of judging whether you made billions for the company will still spit out compensations of millions.
As for startups, I was talking specifically about the founders, not employees. And yes, I agree that startup's succeeding, at least on the Whatsapp scale, is very similar to a lottery. But smaller startups/consultancies can make much more money for the programmers who start them, who have skills that the market cares about more than just programming well.
Small sample size. These types of developers are very few and far between. I worked with one that I keep in touch with and he is making somewhere between 400 and 500k at his current job. His output of good code was pretty amazing to see. He did manage to produce the work of 5 average Bay area engineers consistently.
What you're describing as a great programmer is the equivalent of a chef in the kitchen, what you're describing as good programmer is a talented amateur cook, and your medium programmers are ordinary people that can cook an edible meal.
We have a shortage of professional developers, and a huge number of people who wouldn't get hired if they had the same skill level in a profession where there wasn't a shortage.
We've put lots of meat in seats to compensate for the shortage of mature, professional developers. Meat that can get away with incredibly immature, irresponsible and unprofessional behavior because, hey, who else are they going to get?
Fuck me, the ones that manage to build only slightly more shit than they break already get labelled "decent" programmers.
And it's self perpetuating, because if you combine the amateur playgrounds of most software departments with the systematic wage suppression you create a career perspective anybody with any sense would steer clear of.
With all those clowns calling themselves programmers, and the rest of us being tarred with the same brush, who the fuck wants to be a programmer when they grow up?
Instead of continuing to follow up with just that one person, I'd like to propose an interesting "experiment" for hiring managers:
Do an Ask HN, linking to a recent representative
job posting you haven't been able to fill
1) the job posting, of course
2) company name, location, salary range
3) are the usual requested perks applicable, e.g. flex time, work from home, etc?
4) what kind of resumes are you seeing? How many? What, in general, has been wrong with those people?
Then I'd like to see HN posters dissect the information. Let's find out if the requirements are realistic. Let's find out if the company's reputation is toxic.
I think the same is true for doctors and dentists. Do very few people want to be company executives or politicians? Is there a scarcity? I don't think so.
Sometimes people are well paid because they are setting their own salaries (many believe bankers have hijacked banks from the shareholders), others are setting the rules within society (politicians), or interpreting those rules (lawyers), or just getting close to those who do (lobbyists, bureaucrats, special interests).
In short the wages of workers are dependent partly on their position of power within society .. of which supply and demand is just one element.
similar: set of waiters who want to be actors is huge, set of known actors that a studio can say 'his face on the poster will bring in an extra 50 million cinema tickets' is miniscule. That's why all the waiters are acting for free in film student projects and Tom Cruise is a millionaire.
If you had said Michael Jordan or Lionel Messi ... but Tom Cruise? Really? Tom Cruise is a perfectly good Tom Cruise (a good actor) but a million other waiters could have been him.
There maybe Messi like geniuses in politics, banking, business but the idea that bankers are all paid Millions because "they are worth it" is to me ... laughable.
They are paid millions because they can get away with it.
Our own experience is that even internships do not always yield good results. And the level of most applicants, many of whom are recent graduates, makes me sad. They manage to find work at some other company in the end, but only because we are in IT and IT is big at our place...
Worked in IT in the 90s for 18 months as IT Manager and well I went bald at 22. Did programing when I was 17 and well I woke up with ideas on how to do to fix my issue. To much stress and to little pay for the jobs.
Don't see the change for a "good" IT job for most people. When students start asking about the industry from people that work it they usually get a negative response and told to do something else.
Some of the AI jobs at Google, yes, they need good people. Amazon needs some good engineers at the top guiding the creation of EC2 architecture. But most jobs ask for a much higher caliber of people than they actually need.
I do CRUD work (at least I think I do - I create and maintain the database and front end for a DNA sequencing centre). I have in in depth understanding of SQL - my database returns some moderately complex results. I have solid database design and optimization skills. I know Django in depth. I know Python well. I have a solid understanding of Linux, and know enough to get Django running on the server. I have read up a fair bit on NoSQL, but have yet to find a suitable use case for it. I can do some Javascript / HTML / CSS.
Bioinformatics on the other hand seems to be considered a lot more brain power, when most of it is just scripts to count stuff an plot it on a graph. Its' (often) an order of magnitude simpler than what I do. The majority of bioinformaticains I know have never used a debugger - so the code can't be too complex - or they are WAY smarter than me.
Or when people talk of CRUD, do they mean the sort of thing you should use MS Access for?
There is a misunderstanding on how jobs are made. Consider what these look like for history of English majors? STEM is in demand if you look at the areas with predicted job growth.
In the same vein, there's a shortage of supermodels who want to fuck obese, unemployed men with halitosis and general poor hygiene.
Can we make this more concrete? What sacrifices do you think is being asked of competent developers? What pay do you think they are offered?
How do you know, objectively? Have you tried paying twice (or more) the current market rate, for instance, and binary search for a point where you have enough happy hires and failed? Perhaps the market rate is artificially lowered.
Edit: Didn't realize you weren't the person I originally replied to. Let's reverse your question. What is the objective basis for the claim that engineers are asked to sacrifice to much for too little pay?
I have one possible answer for this issue: Well, no one in the Valley (to best of my knowledge) really advertises that they are going to pay a meaningful multiple of what's considered the market rate to potential employees. So from the potential employee side, it is reasonable to assume that they are going to get paid around the "market rate".
The Valley seems to be at a point where "starting/not starting a startup" is becoming an IQ test (compared to "market rate") financially if you think you are reasonably good at doing stuff, so it is natural that good engineers wouldn't go and apply for a job, because the expectation for compensation is unlikely to be satisfactory anyway (or at least that's what they believe and prevents them from trying).
>And you dodged my questions.
Oh, I am not the OP and I'm not pretending to have the answers. I was genuinely interested in knowing how you measured that you are paying very well, except for the fact that they were competitive with what's considered the "market rate" which is the questionable piece of data here.
Another interesting wrinkle to this is the fact that hiring someone at a high salary is a big gamble. How confident can you really be after interviewing someone?
I'd love to see one of your job postings. I'd bet at least one of the following is happening:
1) unreasonable academic requirements
2) unreasonably broad list of required previous experience
3) unqualified HR people pre-screening and discarding resumes
You need to be willing to "kiss a lot of frogs", i.e. do phone screens and onsite interviews, of people who might generally be suitable. Then be willing to hire not-quite-perfect candidates who are willing to learn and are willing to work hard.
> How confident can you really be after interviewing someone?
I've seen an increase in contract employment postings. That might be a way to go. Start someone on a six-month contract. You might find some takers. Not everyone is ready to "get married" after a single days worth of job interviews.
2. glassdoor.com
3. my network of people who work at various companies and will tell me what they make.
I'm assuming that you are a small company that doesn't actually show up in either of 2 or 3.
What do you think is more likely, that no one is doing it because it's highly effective or that no one is doing it because it causes problems?
I've found that most practices that make for visionary, distinctive companies are both highly effective and cause large problems. For example, Google has a massive shared codebase, does promotion & performance reviews via peer feedback, and shares information very widely inside the company at the expense of being a black box externally. Fog Creek makes all salary data public within the company. Palantir, I've heard, pays all engineers the same (relatively low) salary but doubles down on perks and work environment. Apple under Steve Jobs let the CEO berate any employee over the slightest product detail. Facebook releases new software via IRC chatroom.
All of these practices cause major problems, but they also fix major problems. Moreover, they're distinctive. They draw a certain type of employee, one who's willing to give up convenience in some dimension to gain autonomy, or excellence, or collaboration. It's often better to be differentiated than good.
I see salary ranges advertised all the time in the New England area so its not impossible.
Like some else mentioned; when I don't see a salary range I assume it isn't great or maybe they don't want to piss off the current employees that are working for below market rate.
Of course, some bootstrapped company, or some company which had minimal angel funding or the like is taking more of a gamble, but everything is a gamble when you have no product/market fit and only 18 months or less worth of cash in the bank.
If your methods don't work - try going to a local programmer meetup. Chat up some of the programmers. Encourage them to apply. I go to programmer meetups as a programmer, and all the time I meet other programmers who are out of work, or who are unhappy at their job and looking to work elsewhere. Most are employed, but there's always a few who are looking.
What is happening is this - in 2008/2009 the economy was tighter, and employers could get by with a CL posting, or talking to a headhunter or two. Now that unemployment rate is less than 10%, you don't have dozens of good programmers checking CL for jobs every day. You have to do a little more work to find people. Just like THEY had to do a little more work when unemployment was double digit.
"we pay well" What is the salary range for this position? If you have already hired into this role, what are you paying them? I'd say a specific number is important here, rather than "market rate" (if you say market rate, It's important to define it with a number).
"we don't expect anyone to work crazy overtime" How many hours, on average, does someone in this role work a week? If you offer vacation, how much time off do you offer? How many paid days off per year? If you don't do those things (many companies don't do vacation time anymore), what are the average number of days take off per week.
"good engineers", "competent developers" How do you define this? This is probably hardest of all, but if you already have someone in this role... what degrees do these workers typically hold? How many years of experience is typical? Could you share specific examples of interview questions, and what kind of response is required for an offer to be made? Could you share an example of a problem solved on the job that represents the kind of work someone like this does?
I haven't seen any evidence here in the Boston area that contract rates have risen since I was a contractor 4 years ago. (I still get calls and invariably the rate is always <= what I got before.) One can surmise that there isn't a shortage in economic terms or that companies are unwilling or unable to pay true market rates. It may be true that there is a shortage of engineers a company can afford but that doesn't mean that there is a shortage of engineers. I.e. just because I can't buy gold at $100oz doesn't mean there is a shortage of gold.