Comment by rprospero | Hacker News Reader

Comment by rprospero | Hacker News Reader

> Those with the skills mentioned in this article could easily ask for several times that compensation in a first-year industry job, and would find themselves working on interesting problems in a setting where their computational skills are utilized and valued.

I have to admit that this one always baffled me from the reverse perspective. I always hear these discussions of the great talent shortage and how desperate people are for software engineers. I hear tales of people taking six week crash courses on coding and getting well paying industry jobs. But I just can't believe them.

I don't consider myself a great coder, but I'll forgo humility for a moment to say that I'm decent. Over the course of my PhD, I needed to:

* Write cluster based numeric code for the university supercomputer

* Reverse engineer an undocumented binary network protocol with a packet sniffer

* Work in six languages in a single day (C, Tcl, Labtalk, Scheme, Labview, Python)

* Write code so low level that I had to debug it with an oscilloscope

* Design and implement a data acquisition system that accepted concurrent input from hardware, network, and and user sources

* Port a Monte-Carlo simulation to the GPU

* Write user facing visualization systems

I guess that what I'm trying to say is that I can easily FizzBuzz.

After my PhD, I looked into going into industry, but I couldn't even get a no-thank-you. People complain about dealing with recruiters bugging them about job possibilities, but I couldn't even get one to take my CV. I relocated across an ocean to an employer who didn't even pay for said relocation partially because they're the only people who even got back to me.

Don't get me wrong - I love my current position, but I've known plenty of academics who were miserable. Individuals who were more skilled than myself and sometimes only making as much as Zappos pays their warehouse staff.

If the talent shortage is that bad, why aren't more of these people being poached? Why aren't we seeing a larger brain drain?

All I have is anecdotes, but I'll hazard some guesses.

The talent shortage has a strong geographic component. In SV or NYC, job search sites (a poor way to gauge actual demand, but whatever) seem to list 1,000s of available Java/C# gigs, with substantial numbers of listing that include javascript. But I would probably lump those three languages together as placeholder flags to indicate "standard enterprise developer" jobs. Most of these job listing seem to fall on the entry-level, with a steep fall-off in mid-level offerings, and a few (mythical?) mega-$$ listings. These types of jobs feel more "credential sensitive" to me - if your resume (not CV) doesn't say Comp Sci or maybe MIS, you probably get binned immediately.

Leave those regions, job listings seem to drop. If we can trust these listings to reflect some level of region-comparable demand, we might guess that in Seattle, WA or Chicago, IL, there is about 1/2 the demand of SV and NYC. Cut it down even more in places like Austin, TX or RTP, NC, maybe 1/6 or 1/8 the demand of SV and NYC? I'm sure we could include a bunch of other cities in those tiers, but maybe those are loosely representative.

I often wonder how much job demand looks inflated on these job boards though. A quick glance at Java in NYC, NY on indeed.com shows a few big staffing companies with hundreds of listings. I usually assume there are bunch of duplicate listings from staffing companies for a single position.

I think a lot of lower-level gigs out there are filled by kind-of-passable candidates not making anything like the SV salary numbers that get thrown out on HN frequently. Among strong engineers I personally know in my region, several are frequently contacted by recruiters for jobs that are, at best, side-ways moves for them (eg doing the same gig at some different place for roughly the same money). A PhD (and maybe even a MS) might scare employers at this level?

The other part of this, especially regarding the original article, is that deep, strong statistical/machine learning/magic fairy dust hacking gigs are much less numerous that the standard enterprise dev jobs and that those gigs are even more restrictive in recruiting nature.

Meaning, if you are really recruiting for a top-notch "data science" person, you are probably(?) looking for intensive credentials or portfolios. So MS or PhD in Applied Math or Statistics with easily demonstrated programming skills or a set of completed projects cast in the "data science" space. And if you're not recruiting at that level, the "data science" label means "query some data sources to generate a tabular report, maybe with some rollups."

A recruiter here in Chicago tells me he's dying for entry level ASP.NET and back-end Java coders. I think a lot of these companies just run these guys through at largely unimpressive wages, run a bunch of candidte through as temp to perm, and hope to god one guy out of 20 isn't a complete moron. I've had the displeasure of working with some of these guys. They have some paper certs and some basic understanding of what they're doing but they have zero big picture and any love of what they are doing. Don't get me started on their work from a security or stability perspective. These guys are just in the wrong field. They heard the siren's song of being a coder, did some coding bootcamp or took a community college class or two, and are shoved into big departments churning out junk code.

I think these jobs echo how a lot of basic IT support staff in the late 90s and early 2000s got work. Companies were scrambling for someone to sit down with end users and explain how Outlook worked, install software, and maybe reboot the occasional server. These guys all ran to braindump sites, got a MCSE or CCNA, and now are lifers at the company they fooled into hiring them.

There's something somewhat sad about all this. These people probably would have been better off in a different field but are now stuck in unpromotable positions because they just don't have what it takes to move up. Our own support guy is in his 40s and is barely competent at providing basic level 1 PC desktop support. He's totally in the wrong field. I'm not saying this stuff is a calling, but in some ways it is. Like a lot of technical jobs, you kinda have to make it your religion and invest a bit of your personal time into it because its so fast moving that if you treat it like a 9-5 desk job, you'll fall behind very quickly.

> Like a lot of technical jobs, you kinda have to make it your religion and invest a bit of your personal time into it because its so fast moving that if you treat it like a 9-5 desk job, you'll fall behind very quickly.

I feel this statement should be emphasized a lot more. The technology scene is changing so fast that you HAVE to invest time in learning new things all the time, or at bare minimum hang out with people who do. I feel that this is kinda the flip side to working in technology: people see the great pay and benefits and wonder why they are being paid so well? Well, its because you're skills are in demand....for now. They may not be very soon, so be sure that your skills are upto date.

> I think a lot of these companies just run these guys through at largely unimpressive wages,

This.

When I graduated with a MS in CS 3 years ago, I interviewed at over a dozen companies. I got offers at all of them. The highest offer I got was for 45k. The lowest was 32k. Outside of SV and NYC and the big tech hubs, wages are terrible. For comparison, waiters at the restaurant I worked at (by no means high end) made up to 40k.

Companies want to pay dirt-cheap wages and get tons of highly qualified applicants and shockingly they just can't find anyone!

That said, three years in and my salary has almost doubled (after job-hopping - which did make me sad, as that first job was a lot of fun even if it was a lot of work), so at least there's that, but that's just about the cap for private sector work in the region.

I have a PhD in mathematics and left academia ultimately to end up as a data scientist. Some people at previous prospective employers felt intimidated by my PhD despite the fact that I took steps to downplay it ("No, you don't have to call me Dr. Maney. 'Jack' is fine."). And I've missed out on a few jobs because of the impression that I'd be too bored.

And yes, I've taken on "data science" positions where I essentially became a chart monkey. That, however, is altogether another rant...

So, here's the deal. I've run and hired data science teams.

The first obvious problem is location.

The second obvious problem is money -- you're probably wildly overestimating the value of math, and underestimating the value of knowing your way, in a non-theoretical sense, around the standard ML and nlp tools, and the toolkits that implement them.

Third, I dump all resumes that say mathematics, particularly phd math, unless there is a very strong indication you can code. The problem is this: most data science, in practice, is at least 50% coding / coding type activities. Data is scattered across multiple databases, requires serious cleaning and merging, requires merging external data sources, etc. I also need at least some productization of the output, rather than tossing a hacked-together R/python script with a pile of un-automated db extractions to an engineer.

So I'm faced with a dilemma: if you can't code, where code means talk to multiple databases, extract data from them in a somewhat self-sufficient matter, and deal with a lot of the other tasks, I either have an incredibly unproductive data scientist or I have to dedicate half to 3/4 of an engineer to you, leaving me with a severe negative productivity increase from hiring you, because I don't exactly have too many available engineers.

Now, as I'm sure you'll protest, you can learn all these things! Yes, you may well be able to (though some math people never get particularly competent at them.) However, that's 6-18 months where you'll be pretty unproductive for me, and I need the productivity much sooner than that.

And yes, lots of companies essentially lie about data science jobs, and they're really data engineering / writing data pipelines. I left a former employer after 8 weeks because of this...

When did I say I couldn't code?

When did I say that I wasn't comfortable munging data from various sources?

I don't know the source of the chip on your shoulder. I have no idea who you think you're responding to when you read my post above, but it isn't me.

So you want help or not? If that description doesn't apply to you, fine. But in the commenters experience, they'd encountered all that.

Frustration can make anybody testy. But unless this topic is a rhetorical question, then it'd be nice to respect honest responses that add to the conversation.

When did I say that I wanted help?

In case it isn't clear, I neither need nor want the kind of "help" that is being offered in this thread, thanks.

While there is a great demand, hiring people is actually a lot more complicated than most would-be employees have an idea of. If you're not from the same country (as you seem to be implying), it is even more problematic. The thing is, most employers are anal about hiring the right kind of person because an employee represents a significant investment: not only do you have to pay for his/her benefits, but also he/she will be writing code that will go into their infrastructure, and they want to be really really sure that the person is good.

Now, how do employers find this kind of person? One way is to go to well known universities; most US universities have recruiting events that are chock-filled with tech companies; and this is by far the best place to get a great entry-level position. Once you work at a known company for even a little while, that experience is a signal to other companies (and recruiters) that you are someone who is hireable and they will then actively pursue you. That is the impression that I got, and it has fairly matched with my personal experience.

Outside of university recruiting events, I think your best shot is Meetups. There will definelty be some of those in any city, go there. Introduce yourself, ask questions, learn news stuff and talk to people. I know this is really really hard for a lot of people with an academic background who tend to be rather shy. But hey, you want to work for a great company with the nice pay and benefits, then you have to step outside your comfort zone.

I cannot over-emphasize the need for networking for advancing your career.

The problem isn't your CV or you. It's that the job your credentials seem to command isn't the type of gig any company is willing to risk hiring a fresh out of college guy to do. your academic creds, while wonderful, are simply NOT real world experience.

Your PHD leads recruiters to think you'll demand 6 figures and a SR position. But your experience doesn't show you could actually do the role you'd be expected to. If a hiring manger puts their signature of the offer letter and it turns out you suck, it looks BAD on them, REALLY bad.

I've done enough hiring of SR folks to form the opinion that the problem isn't lack of talent, it's lack of provable talent. As a hiring manager, in most companies, it's better to leave a role open than hire someone that sucks or a chance of sucking.

The fix on your side is the path almost everyone I know that's in a high level position has taken, startups and small companies. You slog it out in an underpaid gig for a few years, think of it as a residency. Then start looking for a real gig.

What do you mean by SR?

Senior. Basically an engineer you can handle a project(1month to 1 year time frame) to and they'll take it from there calling in resources when necessary.

I think it's less important what you've done and more important how you've done it. Replace all this with a single well-written, well-documented open-source project that people actually use and your situation would be different. I'm not trying to diminish your accomplishments, but rather diminish how much the industry values this kind of stuff. The problems you typically work on in industry are much more cut-and-dry, but the important part is not getting a solution in the first place, but a maintainable, well-tested one. Without making too strong of a claim as to whether it's correct, there is a strong negative association for academic code in the industry. I think there's actually a lot of validity to this criticism, though: for each of the projects listed, did you maintain the project at all? Were there other users for the code, or was it just you (or your research group)? This is a place where the academic experience is lacking as preparation for engineering work.

That sounds pretty darn impressive. Maybe the PhD made you look a bit academic? Where did you come from and where did you move?

You either weren't looking hard enough or looking in the wrong places if you think there aren't many companies interested in your skill-set.

Send your resume to the email in my profile, if you'd like. We're always looking for guys like you and we can most certainly put those PhD credentials to work.

Those are not the things that most developers do. When I'm hiring you, I need to know that you can write clean, maintainable, well tested code. We interface with SQL, web services, messaging, rules systems, etc. IE, the majority of development is not "low level", it is "high level".

My company (Red Hat) is always looking to hire. Yes, there are lots of developers looking for work, but lots of them just are not very good. The shortage is a shortage of capable developers.

Reply on news.ycombinator.com