Hacker News Reader

The rise of Medium thought pieces/MOOCs has created the conception that data science jobs are a 40-hours-a-week Kaggle competition, whereas the reality is much, much different/less exciting to write thought pieces about. (I wrote a blog post last year about that phenomena: https://minimaxir.com/2018/10/data-science-protips/ )

That is not limited to data science. I remember the horror of an intern when he realised that real work wasn’t at all like solving neat little puzzles on HackerRank. He discovered that he didn’t really want to be a programmer after all...

I mean, hackerrank puzzles IS how most companies interview so it isn't unreasonable to assume that that's what the job is like. Presumably orgs would ask questions related to the problems they are solving during an interview

It would be a true horror if real work is like solving hackerrank puzzles --- everything is predefined and you're given a set task for a set amount of time, surely a robot could do the same?

To be fair that is the same with many jobs. It always sounds like you will be building the next great thing from scratch, but end up spending far more time fixing other people's crappy code.

Ask any Postdoc or PhD candidate :-). Even they aren't spared from the "discover the next great thing" phenomenon. Perhaps the best thing for students/new employees to realize is that while they may be on the path to building the next great thing, that path is full of potholes and grunt work. Will save a lot of disappointment when they hit the road.

True, although data science seems to have an elevated discrepancy of expectations vs. reality.

>as an outsized interest in mid-level software engineers wanting to be data scientists, mostly because the career development prospects at that stage are grim

What are you basing this on? Senior software engineer jobs are a lot easier to come by than data scientist jobs and from what I've seen, pay better than the average data scientist job as well.

I am just going on my experience, but according to glassdoor data scientists of the same years of experience earn about 20k more than senior software engineers here in Boston.

Could be really depending on location.

Current Software Engineer, for full disclosure, pondering how to move away from the prevalent web dev job market where I currently reside. Data science seems to pay on par.

Glassdoor is not accurate

source?

Depends on the company. I fought the senior engineering leadership and HR to get data scientists on the same scale as engineers (they had previously been lower--much lower).

Even if that's true, the OP was talking about mid level software engineers, so they would be leaving a field where they had 5 or so years of experience and moving into one where they had 0.

I agree with you, but I think data science has the perception of more money and prestige. While senior software engineers often get I paid more, I do believe that data scientists have more opportunity to be involved in strategy and business decisions, which can help one get more exposure to high ranking employees. But this is not always the case.

Data scientists do have opportunities to work with management, but the realities of working with MBAs can be surprising. I found myself reading corporate finance textbooks to be able to fully participate when I started in data science.

This is truer outside of the Bay area- non-tech focused companies that see IT as a cost center, but have somehow decided they need to do 'Big Data Analytics'.

It’s the “it” thing right now. Everyone wants to “leverage” their data and projects in that space get readily funded. It’s also seen slightly better than the typical IT-is-cost-center mentality because of the immediate potential benefit or the risk of losing to a competitor.

OP had I right in the first sentence. Senior Analysts not Senior Software Engineers.

I went into Data Science after grad school 5 years ago.

If it had been as difficult then as it is now I would probably have chosen software engineering.

Honestly, I feel software engineering might be better anyway as it's much easier to demonstrate value building features and shipping products rather than endless analysis and questionable models.

I used to work as a data scientist. This title is not what it should be, I witnessed first hand how business thought data science is the magic that they would be able to rely on to consistently deliver impossible work. There are so much junk that gets shuffled our way and have the expectation that gold be coming out of them. It feels like the financial bubble of 2008, except it's with the inflation of the position and the 'clout' of data science.

There's nothing wrong in data science itself, just like there's nothing wrong with mortgage. But the current trend of software engineer/ non-software engineer moving into data-science is not sustainable. Things will break before it's fixed again, I've always considered myself a software engineer first and foremost, just with some extra machine learning/stats knowledge, and I'm glad to be out of that position now as it looks like we're in for a reckoning soon.

> data science usually means a pay bump

That’s interesting. I’m at one of F/G and a lot of the data scientists want to go the other direction to software engineering because we receive about 60% of the RSUs that they do. A few people on my team actually did switch; they said they found the data science work more interesting but an additional $40-100k per year can make a really big difference over the long run.

I think this is dependent on what someone means when they say "data scientist" (mentioned as Type A vs Type B in the article).

Facebook and Google "data scientists" (meaning those who hold the title) are really more like analysts -- they analyze data to inform decisions and use a lot of SQL. They make prototype models (usually based on less cutting-edge techniques) that get passed to engineering teams if they become worthwhile to scale/formalize. These folks get paid less than SDEs usually.

The other type of "data scientist" is basically an SDE (maybe SDE-lite) with research-level ML skills. These get paid similarly (or higher in some cases) than SDEs. I believe Facebook and Google call these SDEs. Sometimes the term "applied scientist" is used to describe these at other companies as well.

I have a research background and do pretty heavy duty machine learning. It’s not analyst work (the internal job family is “applied scientist” while the external facing title is “data scientist”). It’s still not SWE compensation. As far as I’m aware, there is only one team of data scientists that makes the same as SWEs at my company.

Does whatever you build go into production or do you need someone else / team “take care of that part”. That is where the “data science is just Statistics” people’s wheels come off when they realise production ML needs senior software engineering background.

Some of what I've built is currently in prod at a very large scale (which honestly is a bit freaky). Depends on the particular project though. Our team very rarely hands stuff off to SWEs (although they frequently code review); for the most part we implement everything ourselves.

Exactly.

At my company, Type A are called "Data Analysts", but at Google Facebook they're called "Data Scientists". Type B are "Data Scientists" at my company, but "Machine Learning Engineers" (or SWE-ML or some other combination) at Google and Facebook.

As a Type B, at my company, I'm on the same pay scale as the SWEs. The Type As are not.

At one of the national labs whose jobs listings I’ve looked at, they have people in ML/data science and ML/data engineering. The first is in the research department and the second is in IT.

I really am surprised to hear this. I'm about 250 all in and I haven't heard of many software devs pulling over 200 all-in. This is for Boston and I'm 6 years out of my undergrad with no graduate degree (although I didn't go to college until my late twenties and I have noticed my maturity helps a little). Maybe on average it's the same but data scientist at the right company has a higher upper limit?

FAANG skews the graph. levels.fyi

200k+ is common at L5+ at my company and it’s not Google or Facebook fwiw.

That's interesting that there are data scientists going the other way - you don't really hear about that on the outside.

What sorts of SDE positions do these data scientists go into? Are there any additional skills they pick up as part of the transition, or are strong Python/SQL skills enough?

"analyst that can build their sql query into a scheduled ETL or daily process of some sort. You work in pandas instead of excel sorta thing"

In my experience from my old employer...clients like Google get billed $120/hr for SQL analysts' services; ten years ago, staff earned $20/hr or a little more, and today, they've replaced almost everyone with offshore employees making $3-4/hr.

Where is the place that mid level software engineers think there are not good career prospects and they'd get paid more? I can only guess it's a place where there aren't many dev jobs. My experience is in the Seattle area and we are begging for people to even apply for jobs. There are 10,000 jobs easily in Seattle. My company would love to grow its dev force 50% and we can only get people by hiring them away from another company (perhaps an obvious comment :-)), and by hiring new college grads.

If your job is working you too hard, with not enough pay, then people here get another job. It seems harder to high people with some experience at my company anyway. New college grads make 120k+ at top companies (we are a startup but not a unicorn, we pay a little more than that).

Pardon the question: What skills are you looking for that define "mid level software engineer"? I'm a long-time engineer at a single company. I've been more and more tempted to strike out elsewhere as I feel like there is nowhere else for me to move into position/pay wise where I am. I need help to determine how to frame the skill level that I have and/or where to focus on so that I can claim/apply with a certain level of skill.

Title placement is often just a rough estimate based on years of experience. If you've been at a single company for a while, the usual advice is to break your time up into roles/projects on your resume.

Thank you