Ask HN: Transitioning out of Data Science | Hacker News Reader

Ask HN: Transitioning out of Data Science

10 pointsdata_says_no8 comments

I've been working as a Data Scientist for about four years now after finishing a PhD in physics. At the beginning there was a good mix of both engineering and statistics, but over time it feels like a lot of the work of "building things" has been shifted to software engineering roles. What's left over is most "Business Intelligence" work, at a larger scale.

Because of this, I'm considering leaving Data Science. I was curious where other people have ended up after leaving and what other paths are open?

View on HN

My prediction is basically once Excel can handle larger data sizes and do basic ML algorithms (like boosting / random forest / etc), and data manipulation becomes more the work of a BA (say, via Alteryx), and they start teaching what basic algorithms do (like Random Forest, Boosting, etc.) in a business analytics course (which right now is basic hypothesis testing / linear modeling) then we will see the death of a lot of data science roles and people will transition into more specialized technical roles. I could see this moving into that direction in say 2-5 years.

I've worked with a lot of Business Analysts and they usually end up fighting over p-values and R-squares, even when the linear models they are creating are total garbage due to overfitting and multicollinearity.

I agree with you that easier to use and ubiquitous data prep+ML will eliminate a certain type of data science role. However, most good data scientists I know work on taking a business problem from no/bad/inappropriate data to the right data to address the use case. It's not just about data manipulation and ML.

I'm in a similar position myself. 5+ years in Data Science roles, and I've realized I'm less interested in the "generating insights" part of data science and more in building systems and architecture that support this type analytical work. This is what I've found in thinking about moving into a different type of role. Some of the following is based on what I've seen colleagues do.

If you're interested in staying technical: Machine Learning Engineer, Software Engineer, consulting, and certain types of Pre and PostSales. A lot of people who work in "Data Science Engineering" aren't coding ML algorithms or doing linear algebra. What they are doing is building the systems that support data science processes and workflows. These are the systems that help to create repeatable insights for broader use, and not just one off scripts. Having a data science background + development skills goes well together for this.

If you're interested in staying in Data Science and/or Engineering, but not having your technical work as a deliverable: Product Manager, Data Science Manager, Engineering Manager

If you're interested in getting deep into modeling: Statistician, Operations Research, and some AI roles.

If you're interested mostly in talking about Data Science and being focused on the business side of things: Sales, Sales oriented Chief Data Scientist, Business Development

If you're interested in talking about data science and doing some technical work: Data Science Instructor, Data Science Product Evangelist, PreSales (depends on role).

High Level Technical Support is an option as well. Not many people can diagnose a bug where some error in a algorithm is resulting in a faulty design matrix, or find a condition that results in a divide by zero error in a complex mathematical function, etc. You have to understand the theory behind a lot of the algorithms for this.

Also, something I've learned. When I interview for roles now I'm VERY honest that I want a significant portion of my job to be coding and serious technical work (I enjoy building a nice Tableau/Spotfire Dashboard, but I don't consider that to be very technical). I don't have a software engineering background so it makes it harder for me to find a job, but I'm also more likely to end up in a job where I'm a fit. I've found these types of blended jobs are out there, but you have to hunt for them and be patient.

I went through this same thought process a few years ago. I basically gave myself a crash course in CS and engineering and "reinvented" / positioned myself as a software engineer.

I left my data science job and spent a couple of years as a "pure" software engineer and now am a "research engineer" which is basically a software engineer who understands machine learning and deep learning.

By and large I'm very happy with the move (other than that the company I quit as a data scientist got acquired and I would have made more $$$ if I'd stayed).

When I read your username I was a little taken aback. I have your python data science book and just assumed you were always a ML Software Engineer. I didn't realize you were a data scientist who transitioned into a more engineering focused role by teaching yourself CS.

I'm trying to do this myself, but it's mostly the road of failures with success somewhere out there. Seeing you did it gives me hope that I'll get there one day. Thanks!

That sounds like an interesting direction to go in.

Do you have any advice on making the transition? Did you stay within the same company, or move? What did you dp to demonstrate the ability to work as a software engineer?

I went from a data engineering role (jack of all trades data piping) into web analytics implementation/marketing analytics (ROI stuff) and now am a product manager. At this stage I don't think I'll go back into Data Science as the barrier to exciting companies with great processes is quite high and I think the field is still a bit too immature to work anywhere else.

Marketing and product teams are quite fun to work in from my experience.

Get a job building ML systems, not one where you are writing queries (though they may be complex) for a business person.

Reply on news.ycombinator.com