Comment by ford | Hacker News Reader

Comment by ford | Hacker News Reader

Parent Full thread

I work in ML - I might make 3 buckets for ML careers right now:

1. ML/DL Researcher

2. Data scientist - 20/80 engineering vs modelling

3. ML Eng - 50/50 (or 70/30) engineering vs modelling

People suggesting working in engineering to support ML are right that there's a lot of demand, but it's not what you're asking for.

Becoming an ML/DL researcher working on novel techniques or new models will be hard without academic research experience. Few companies are big enough to support true research, and the ones that are have a very high bar even for people with PHDs

What I call "data scientists" apply math/ML to real problems. The people I see here have a quantitative background like physics/math/CS. Often they have more general quantitative skills that go beyond ML. People like this will might work on things like fraud where an eng pipeline exists and small improvements in the model are valuable.

There are more of these roles than "true research" and they exist at small companies because it's applied. You can get into this with demonstrated evidence in side projects + a convincing background, but professional education might be the most sure way.

Finally - there's a lot of demand for engineers who can do both modeling and the requisite engineering. A model is a small part of what goes into a production ML feature - you need a data pipeline, automated retraining/prediction, a place to deploy the model, monitoring on eng stats + data stats, and the usual application backend/frontend to do something with the results.

You might be able to get into this with some demonstrated experience in side projects assuming you're a SWE already, and depending on your standards for where you want to work.

> Becoming an ML/DL researcher working on novel techniques or new models will be hard without academic research experience

This is not correct for current DL research. I know many undergrad engineers who wrote papers in top conferences. Current DL is mostly about implementing ideas, running experiments, having good sense of data etc. rather than theory. It's an open secret in DL that theories are just there to please reviewers and mostly gibberish and often time plain wrong. e.g. batch norm paper, where what they theorised about it was proven not just false but completely opposite. Still batch norm is heavily used because it works.

I didn't mean to imply it needed to be _graduate_ research - however it would be news to me if they were publishing at top conferences independently of a lab or research branch at a company.

How do the people you know go about it?

I didn't said they were publishing independently, just they did not had any academic research experience. e.g. in FAANG, it's not hard to get into teams which publishes ML papers. Also, I know one guy who contacted one professor, and they collaborated and published few papers while doing full time engineering job.

I think you're talking past each other -- doing solid ML research vs being well paid as such a researcher.

I would add a fourth bucket -- ML Ops. Operating an ML based system is different enough from other systems that I'd consider that it's own specialty.

Could you please provide some examples of what a ML Ops person might do/manage in their day to day jobs?

- Tracking changes is different. Not only do you have to track code changes, but also training data and model changes. You need to build systems that allow for this change management.

- Monitoring -- you have to build specialized monitoring to check for model degradations. Is the model still outputting valid/correct predictions? If not you need to roll back to an old model (see above).

Those are the two main ones.

I would say this falls under #3 - ML engineer

Oh wow, those are actually fascinating problems! Thank you.

Would you (or somebody else) mind comparing/contrasting Data scientist/ML Eng a little more? I'm not sure I understand the difference (and perhaps like many roles/titles in our industry the line is blurry).

Never mind, I mentally flipped the numbers. I read 20/80 and 30/70 but it's actually 20/80 and 70/30. IOW, Data scientist spend a lot more time modeling, and ML eng spend a lot more time engineering. makes a lot of sense. I'll post this comment anyway in case it helps someone else.

I think of ML eng as more infrastructure and scalability. Possibly doing tasks like converting lab models into models that can be run at production scale. There is a blurry line between the two because it makes sense for some tasks to have shared ownership - just like you tend to have with people reaching across the stack to get something done in front-end vs. back-end web roles. As with anything, as you get more experience you get more comfortable jumping around and maintaining a larger set of concerns

Is it possible to do 3 but stick to coding and not touch any dev ops work?

My nightmare is finding a role like that and realizing I’m just a dev ops guy.

If by devops you mean managing architecture + deployments vs only writing code, I imagine that exists at places with mature infrastructure, but honestly MLOps is developing so quickly that small to medium ML teams will go through some infrastructure churn.

The best advice I have is apply for jobs with descriptions that sound like what you want to do and clarify with the hiring manager/recruiter that it actually matches what you're looking for

Reply on news.ycombinator.com