How do you break into a career in machine learning? (2020) | Hacker News Reader

How do you break into a career in machine learning? (2020)

theexclusive.org

56 pointsgreghn63 comments

I moved to a Machine Learning Engineer role a few years back. I just did an internal transfer to a ML team with no previous knowledge. The team trusted me to pick up whatever I needed on the job.

Now I interview tons of people for MLE roles. Trying to get in "through the front door" is incredibly competitive. Not only you have to prove you are a great engineer, but also good ML knowledge. You'd be competing with people with MSc and PhD.

Whenever possible I still recommend following the path I did: become and excellent engineer, and then look for an internal team to transfer to. Any sensible manager will take you in and help you to grow into the role.

As always YMMV. This is my sample size of 1.

Personally the path of becoming an excellent engineer is the best path (possibly hardest) to take on any situation, precisely because of the versatility in scenarios like this. I'm by no means an excellent engineer, but I like to plan and approach opportunities thinking someday I'll be one.

Starting the path is also age restricted. I've had some friends ask how they can switch into the industry by asking what languages they should learn. It's a hard pill for them to swallow, that they're better off becoming good at programming and great at IT, rather than trying to start from scratch after 30.

Your last sentence is confusing, rather than starting what from scratch after age 30? An ML career or trying to become a great engineer/programmer/IT person?

Great engineer like the poster above mentioned. Switching from web apps to ML isn't that big of a jump.

I moved to my current team in my mid 30s, so there is always hope :)

As a Machine Learning Engineer what tools do you use on a regular basis?

Spark in Scala, Pytorch, Docker, bunch of infra... We run in AWS and have a ton of proprietary tools.

thank you.

> become and excellent engineer

Is there a reliable definition to that? A good balance between technical and communication skills or something among the lines?

> Is there a reliable definition to that? Not really, much of it depends on the group culture.

However in general this should be enough: Demonstrate you can do the work by "being smart and getting things done" and by not being a sociopath. If you are an asset and not a liability, you will become an excellent engineer.

Are these academic types great engineers? Do you find that the academics have a major edge in domain knowledge over self taught programmer?

You're not really an academic unless you're working in the academy, right? Once you move to industry, you're a professional.

Be consistent, a move to industry makes you industrial. It takes mere oath to profess.

Usually you can tell people who are purely academic. They excel at domain knowledge but are almost naive about building real world systems. They might come in as juniors. I don't think we would loop anyone as mid level or above unless they have experience delivering working systems in production.

Personally I don't mind the candidate's background so far as they can do the job (how the interview is a sucky way to determine this is another topic of conversation).

At the end of the day, it is about engineering. I'm self taught as a programmer, and I have also translated the latest deep learning paperwork into live code, AND i have novel research and experiments in code repos (not papers).

The thing lacking is a hacker's credibility as an ML researcher, which i think is ironically preposterous given the self-teaching of complex domains with engineered proof.

I don't see what an PhD has on the class of self taught hackers is what I'm trying to say. Just kidding, I know the answer is that they are trained in a certain institutional mentality. Give me my downvotes.

I lead a Ml Product team and in my experience no they’re not. The academic types are very proficient at research and exploratory data analysis. They’ll whip up a Jupyter in no time but they struggle understanding a live production system.

This is where an ML engineer comes into play, they’re not so academic but way better at writing production code

What of the engineer, who mastered a more complex domain on his own, perhaps innovates in that same domain regularly, do you suppose he can do the academic research?

Spicy comment >_>

> Now I interview tons of people for MLE roles.

I come from fintech, what suggestions do you give to a former dev re-entering tech via AI and ML and wants to focus on the Product Dev side of things?

I managed/collaborated with a team of 3-4 devs as a co-founder during it's peak and then did a dev and consultant stint at a mega corp after getting fed up with how bad PM can ruin everything and self-sabotaging itself.

I'm now studying a BSc in AI and ML to get back into tech and I've realized that my strengths won't be as a developer and would I'd prefer to focus on being a much better PM than what I had.

I'm afraid my expertise is interviewing devs so I can't help you there. I'd also need more details to give more tips.

From the sound of it I'm not sure a BSc in ML will land you a PM job. I can also say that we are an applied science group. Our PMs are experts in our specific domain, not necessarily in ML.

Get a job as a data engineer on an ML team. Build some credibility, eventually ml engineering tasks will flow your way. Study a ton outside of work, and then apply for a proper ml engineering jobs, embellishing the achievements at your current role.

What do DEs do on an ML team?

Fake it until you make it. If anyone catches you, tell them you're building a career model.

"I was just hallucinating new data points!"

"They asked me how well I understood theoretical physics. I said I had a theoretical degree in physics. They said welcome aboard."

Personally I’m moving out of the field into general software development. As a data scientist I never found myself getting into a “flow state”.

Most importantly I’ve found the impact of machine learning to be limited outside of massive companies which come with their own headaches. To add to that the number of jobs is limited and the competition is fierce.

Not to say any of this is a negative, I just recommend people only get into it if they genuinely find working on ML problems exiting enough to do in your free time.

If anyone has done something similar, I’d love to hear about it. Most recruiters seem to be pretty surprised I’d want to do this.

Same here, I did a bachelor in CS/math, then a masters in Data Science. Really enjoyed the theory/application a lot but did not like the idea of having to do it as a living.

If I have to build a piece of software then I can be quite certain that I can deliver. For a Data Science project on the other hand, there are a lot of ifs, e.g. quality of data, how well does it actual generalize, etc.

I think in an environment where the higher ups understand ML well and you have a good team it could be fun; the moment the higher ups don’t understand it so well, I feel like it could be the source of a lot of stress.

My recent recruiter was also skeptical at first when I applied for a swe position with an ML background.

I did and now I'm trying to break out of it.

Into what?

I want to do backend development next, I find it much more interesting (and more rewarding to work on based on the things I've done).

This is literally opposite of me. Would you mind elaborating a bit on the negatives of working on the ML side of things?

Definitely not saying your wrong or that it's better than backend dev (it's probably just personal preference). But as someone considering it, I'd like to hear the good and bad of each type of role.

I too moved away from ML after actively pursuing it for many years. YMMV but here are my reasons

- Scientists dont always make the best 'clients'. The requirements you spend months implementing may be completely obsolete by the time you are done and then completely unused. - You often dont understand or are made aware of the impact of your work. - Its challenging to compete with Masters/Phd graduates who have spent years delving into ML. Entry-level knowledge only takes you so far. So its more likely that you wont work on cutting edge ML research. - MLE work in my experience has been mostly around infrastructure management and data security. Again it has interesting challenges and hard problems to solve but with the speed of the AI world, it all boils down to facilitating the scientists and researchers as much as you can

Out of interest who led the team? Did you have a product manager? Ideally they should make everyone aware of the value of the work

We did not have a product manager. In my team, there was frequent churn at the manager position. Which should have been a clear indicator that my division had no clue what they were trying to do.

I was naive and trying too hard to stick to ML but lesson learnt eventually.

Thanks for sharing your input here.

I ended up doing the same thing and moved to backend development at the beginning of the year. ML engineering always seemed very mystical to me but after doing it for almost 2 years I got burnt out very quickly. Partly due to some incorrect expectations I had from the role, and partly due to the work I was given either being not very interesting or very difficult to measure (in my opinion and experience at one company, of course). Backend devlopment felt much more rewarding with more concrete goals and strategies which I heavily preferred (not saying all ML engineering task are shooting in the dark, but this was very much my experience where I worked).

If you are in undergrad, I'd recommend something like,

Math or math-heavy science BS -> undergrad research -> computationally heavy PhD -> entry level DS or ML engineer job -> senior ML job (within a year or two)

If you are older and looking to pivot, I'd recommend, Data engineer -> senior data engineer -> entry level DS -> senior DS

You don't need a PhD for DS/ML Engineer. Even at FAANG, even in their research labs.

Usually a PhD is only in the requirements for Research Scientist (RS).

That said, I did a PhD (and am now a RS). It's a fantastic opportunity to learn fully focused during a few years.

But if your sole objective is the career (which is ok!), don't do a PhD. There are much easier ways to break into ML industry.

To add to this, companies at Google-scale tend to have a huge variety of ML related jobs, ranging from low level things like optimising libraries for different hardware, to the more general research positions where people are working on their own pet projects. Plus everything in between - data management and curation for training models that get used in production, people who try and figure out how to productionise cutting edge research, people who build the infrastructure that other ML engineers use (and here again, everything from hardware/server people, cloud, site reliability, tooling) and the list goes on.

I know of at least one person who got an ML job at Google, but didn't apply specifically for it. They had a very strong ML background and applied for a generic software engineering and got team matched. That seems like a reasonable way to go if you don't want to go through a research interview loop.

I;'d like to echo this. I learned a long time ago that I don't want to be a "machine learning engineer"- I have no interest in designing new networks, feature selection, or training as a daily job. I know how to do all those things but it's not somethign I pursued at Google. Instead, I found jobs where I could work with those people (often the ones doing the real state of the art research at scale) using my experience, in ML, data engineering, pipelines, and HPC.

There is nothing quite like having a world-class researcher ask you to figure out why their model is exploding, and tracking down the crazy things that happen on TPUs when their math isn't absolutely perfect, then helping them fix it, and see them publish their results (or put them in prod). Or knowing enough software and hardware to debug a tensorflow TPU problem with an oscilloscope connected to the voltage regulator in a hardware lab.

Personally, i gained these skills over a long period starting in the mid-90s (working on machiine learning, and then later HPC for biology, and ultimately back to machine learning). But I am a slow learner. probably the shortest path is to get accepted to a major university and do really well in your ML and CS classes, then parlay that into a job in a FAAMG, then figure out what you want to do with all your skillz.

I got a unicorn senior RS offer without a PhD from a company that had mostly former FAANG top brass after interviewing without knowing it was for RS, I thought it was DS/ML. I declined because of "culture fit". Everyone has a PhD, they assumed I had one, I don't even have a bachelors. We still hang out and laugh about it.

I had been working in Attitude Determination and Control and Optical Systems Engineering for seven years before that interview and I just like, knew the stuff from the job. I've been back on pure-SWE roles for four years already and I don't think I could do it now. I have the intuition but I couldn't white board proofs for tree based algos and manipulate integrals like I did on that interview for sure.

> entry level <whatever> job -> senior <whatever> job (within a year or two)

Do words even mean anything anymore?

I think most of the comments assume that machine learning engineering equals machine learning. This is always not true. If you are an MLE, chances are that you work on the infrastructure and data pipeline side of things rather than on the research. If you are lucky, you get to productionize a prototype, which might involve re-writing it, or you might get to build an in-house hugging face. You absolutely do not require a phd to do this.

Then there are these other roles which involve prototyping new ways to train a model, or taking a paper from 5 months ago and see if it works for your use case. Or you know, just work on something that you can eventually publish. At FAANG, these are usually the "Research Scientist" or "Applied Scientist" roles. Most of these require a phd, but it's completely possible to get an offer with just a masters (I did), and I know of at least one case where the person "only" had a bachelors (and some experience). But by far the most straight-forward way to break into these roles is to have a phd.

Do you really want to, or do you just want lots of money?

ML jobs are split into theoretical and practical.

Theoretical involves building proprietary models based on academic papers, and training them. This is where the PhDs are going.

Practical involves deploying ML models, either in the cloud or on devices. This doesn't require the heavy theory that is still rather new in university, it is more about application programming.

The theoretical jobs pay a lot more, the practical jobs just require a cursory knowledge of ML, and not the nuts and bolts. The latter requires a lot more patience to understand the explosion of inference hardware (esp. Nvidia's convoluted tooling).

Theory: can't unless you have a phd.

Practical: learn python and tensorflow, C++, and devops.

MLOps is a good way for someone with a Cloud Engineering or DevOps skills set to join a team and start learning from within

In my experience no matter the field breaking into it via ops is almost a certain way to never get to where you want to be. Cloud Engineering/SRE/MLOps are so in demand a company would be foolish to let you move up unless you really, really made a stink about it.

Better to just get your MSc in statistics/CS. It's possible to break into the field with less but of the (very talented) ML engineers/scientists I know the ones with the BScs are basically stuck. Most people want to actually make cool models and novel ideas. You won't get to this position without an MSc /PhD.

>In my experience no matter the field breaking into it via ops is almost a certain way to never get to where you want to be. Cloud Engineering/SRE/MLOps are so in demand a company would be foolish to let you move up unless you really, really made a stink about it.

You need to pick better companies. If the place you work handles employee growth and development by saying this employee is to valuable to support their career then gtfo.

If you've got an interest in ML and you're also interested in embedded/low power devices then we're looking for this rare combination. Preference is for people who are in the UK due to admin overhead.

My email is in my profile.

Funny there's no mention of ML without a PHD. Anyone done that?

I’m still very early in my career, but I went straight from my bachelors to working in ML Engineering at a startup. I think it depends what type of ML job. If it’s heavy in research, I imagine a PhD would be much more important. We’re doing some research in terms of building some new models, but good portion of my job is on the infrastructure, pipelining, side of things.

I'm a Data Scientist who builds ML models. My bachelor's is in psychology, I just studied and learned how these algorithms work.

It will be interesting if you can share how you landed in your first ML job, once you learnt the algorithms. I think getting the first job in ML role, if you don't have formal qualification in the related field is the hard part.

Got my foot in the door doing an after-school program teaching kids to code.

I leveraged that to get a teaching assistant job at a bootcamp for adult professionals.

I networked my arse off at the teaching assistant job until experienced programmers (such as instructors) realized I knew my stuff but was underemployed. I got a couple of side gigs doing BI Analytics that way.

After doing this, I had a tough set of interviews for my first full-time role. Every failed interview taught me about my weaknesses and blindspots, and I learned from them. I opted to get stronger at system design, stats & ML algorithms, though I feel like grinding leetcode could have been another approach at this point.

Because I had a wide set of marketable skills within data-oriented work, an analytics consulting firm took a liking to me. I had versatility for billable projects, and I got a bunch of tech certifications in AWS/etc. This role would be describable as 'Analytics Engineering'.

They overworked me for a little while, then my next role was a Data Scientist role that was on my own terms.

I don't want to make it sound like I could just jump in no problemo. I had to think strategically about how to climb each rung of the ladder. But I am now at a point where I have the experience needed to be a senior. While some companies might turn me down for not having a piece of paper, there are enough who actively want me that I am sitting pretty with my career.

Depends what you mean by ML; there are a lot of successful ML practitioners with no formal ML education, but far fewer (but not zero!) ML researchers without the expected academic background.

Finding a small, contained use for ML in your software/data job is a good path into the former, but I have no advice on the latter.

There's no mention of it because the article seems to be focused on research, not necessarily on applied work (for which you don't necessarily need a PhD).

Comma ai seems to be doing fine and I don't think anyone there has a PhD.

I run several software teams that all have ML engineers on them, and two ML-specific teams of all ML engineers. Perhaps 14 ML engineers in total across my teams. We pay FAMGAN market rate. Their experience ranges from senior ML (6-8 years exp) to three who I hired right out of undergrad/masters, and two of those were interns who got job offers after internships.

Only one out of all of those have a PhD, and it’s only pseudo related. You absolutely do not need to have a PhD for 80% of positions in the modern world of ML in my experience, and I’d go as far as saying unless you want something significantly more prestigious than “market rate ML job doing interesting work” then a PhD is probably a net negative in life as an ML person given the opportunity cost. I have definitely turned down prestigious academia PhD types who wanted to move to industry in strong favor of strong SWEs with practical ML experience, and have a strong preference for same.

This definitely isn’t the answer academia or most people who have sunk cost of their time into PhDs would agree with, or necessarily like, but from a practical perspective it’s my experience across much of industry.

The attitude that you need a PhD to practice ML is over a half-decade old.

Modern ML tooling has progressed enough that not only is a PhD not necessary, but overcomplicating ML model construction fully utilizing said PhD can easily lead to technical debt and make things worse.

I got a job as a junior ml dev at Coveo's R&D (through HackerNews no less) with an MSc in Experimental Medicine (BSc in Biochem before that). I was sure they'd never take me but they did. I think the main reason I got hired was that I did really well on their interview take-home test. I had a great time their but ironically, I left to start a PhD in deep learning as applied to biology so that I can strengthen my theoretical skills. No regrets, but I'm sure I'd have been able to grow at Coveo without the PhD.

I made a video about this: https://youtu.be/YcJN_ZiFw9w

Has anyone tried asking GPT-3 or one of its cousins?

I used gpt-3 to make a machine learning curriculum and study guide

https://twitter.com/bsunter/status/1535730704138444801?s=21&...

Reply on news.ycombinator.com