The Machine Learning Software Engineering Interview
eng.lyft.com
eng.lyft.com
This is a symptom of "bullshit" going on around in big tech companies. "bullshit" here is an economic term defined in the book "bullshit jobs". https://www.amazon.com/Bullshit-Jobs-Theory-David-Graeber/dp...
Reading through the post, I was noticing
So much corporate Jargon which really does not mean anything important.
Dehumanizing language when describing people interviewing and being interviewed and its process.
Too much obfuscation of ideas that can be very simply explained.
glorification of simpler problems into heroic challenges.
Delusions of Grandeur.
Today's such jobs are tomorrows layoffs.
I think I will stop here. I have crossed my negativity threshold for the day.
As a perfect example of this is the trend in most places I've seen where data scientists strive to increase the complexity of their model (so they can prove how "smart" they are). A huge part of a software engineering education (whether in the classroom or in dev shop) is learning that complexity is the enemy. No engineer would choose a 3 layer MLP over a simple linear regression for an imperceptible improvement in performance.
The additional irony of all this is that a decade+ ago a software engineer who had strong quantitative and numeric programming skills was rare and an elite find. You would have thought that the data science boom would have dramatically increased the number of these people but I find them even rarer.
I think some of this is exacerbated by modern pillars of machine learning and data science. Competition sites like Kaggle are entirely based on maximizing test set accuracy, and so winning submissions these days are huge morasses of ensemble methods that are trained for days and weeks on GPUs, but in the end they are often only marginally better than some of the fairly basic standard approaches. And when companies like Google are building their bots for Go or Starcraft, they are using cutting edge techniques. When people see that and get inspired to get into data science, thats what they want to do, even the the majority of problems are more rooted in data quality, thoughtful understanding of the problem, and more rudimentary methods.
Its also the result of some of the rhetoric of important figures in the field. Yann LeCun has pushed back strongly in the past on criticisms of modern day machine learning's occasionally lack of concern with introspection and model understanding. Judea Pearl, a Turing award winner for his work in machine learning, devotes large portions of his pop-sci The Book of Why attacking the field of statistics on the whole, as well as engaging in multiple attacks on historical influencers in the field with such ferocity it borders on character assassination. He has even rebuffed modern critics, such as the very widely respected Andrew Gelman, by saying they are "lacking courage" by failing to accept his "revolutionary" causal inference methods over the traditional ones used in statistics.
The attitude is driven a lot by the people and institutions at the top, and as someone in the field, I unfortunately encounter this kind of thinking way too often.
Due to the hype it becomes a goal in some organizations however. "We need to do machine learning because we have big data" or some such. Doesn't matter if the problem could've been solved in 5% of the time and cost with 20 lines of code, thou shalt use machine learning.
It doesn't help that data scientists (creating and training the ML model) and software developers (creating and maintaining the software) usually come from different backgrounds, requiring a "data engineer" as an additional intermediary.
It always a problem with hype, blockchain (or merkle trees) has the same problem but worse, because the problems it solves well are rarer and more narrow.
To put this statement into context, I'm speaking as someone who had been writing code in C, from the era of PC XT. Perhaps NIPS 2010 was a rite of passage to ML for me. There is a screen, full of industry grade C++ and PyTorch in front of me, right now...
Yes, I know that there are folks that deal with vast amounts of data with inscrutable relationships where you need fancy algorithms to make progress. But seriously, most problems just don't need it, and many folks would be better off with mastering basic statistics and data analysis.
It's fascinating how far you can get with basic stuff. My favorite? Statistics for Experimenters, by George E. Box. It's like a secret weapon! https://www.amazon.com/Statistics-Experimenters-Design-Innov...
Were you around during the dotcom era?
Although I'm not old enough, I've heard that OR in the 80s was the same crap.
The problem could have been tackled with greater accuracy using machine learning, but it would have taken a long time for the system to generate enough data points for a sound model and would have required more storage space. This was also complicated by the fact that the model had to be regenerated whenever the physical system being modeled was changed.
The linear programming solution was a lot cheaper and was "close enough" to serve as a useful approximation.
https://en.wikipedia.org/wiki/Operations_research
Basically a mathematical approach to problems of logistics and scheduling developed first in WW2. Very powerful in the domains for which it was developed but less generally applicable than enthusiasts hoped, leading to the usual “hype cycle”.
If you have a problem OR could solve or just want to fool around with it PuLP is very easy to use https://pythonhosted.org/PuLP/ Of course the ease of use means that it is a commodity skill now.
That said, SpaceX interview process is even more ridiculous. The first step is to talk on the phone with a non-engineer recruiter who has to ask you highly technical questions, but doesn't understand a word of your response, and you know it. They then sort of have to correlate what you're saying with the answers they have and decide whether you know anything or not. The most uncomfortable interview situation I've ever been in. Or at least that's how it was a few years ago, maybe they've changed it. I was so thrown off by this, I totally fucked it up and never got to the second step, in spite of nominally having all the right experience. To relate, imagine trying to explain low level assembly to a five year old, over the phone.
And, these guys aren't even Waymo.
Bullshit is neither an economic term nor an anthropological one. David Graeber is an anthropologist, not an economist, though he has written inexplicably popular books on economic topics that betray his lack of understanding of economics.
Bullshit is actually used as a technical term in philosophy occasionally.
http://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on_...
> One of the most salient features of our culture is that there is so much bullshit. Everyone knows this. Each of us contributes his share. But we tend to take the situation for granted. Most people are rather confident of their ability to recognize bullshit and to avoid being taken in by it. So the phenomenon has not aroused much deliberate concern, or attracted much sustained inquiry. In consequence, we have no clear understanding of what bullshit is, why there is so much of it, or what functions it serves. And we lack a conscientiously developed appreciation of what it means to us. In other words, we have no theory. I propose to begin the development of a theoretical understanding of bullshit, mainly by providing some tentative and exploratory philosophical analysis. I shall not consider the rhetorical uses and misuses of bullshit. My aim is simply to give a rough account of what bullshit is and how it differs from what it is not, or (putting it somewhat differently) to articulate, more or less sketchily, the structure of its concept.
"Debt" I think shows a deep understanding of the relationships economics has with history, philosophy, and society. Graeber knows he's not an economist but he's got a point to make and he's not shy about making it even though it says less than flattering things about some aspects of economics.
You're link is broken for me btw.
On Bullshit
https://en.wikipedia.org/wiki/On_Bullshit
https://noahpinionblog.blogspot.com/2014/11/book-review-debt...
> Now, this may sound a little silly - if someone wrote a book called "Metal: The First 5,000 Years," and then filled that book with stories of war and bloodshed, never failing to remind us after each anecdote that metal was involved in some way, we might be left scratching our heads as to why the author was so fixated on metal instead of on war itself. And in fact, that is indeed how I felt for much of the time I was reading Graeber's book. The problem was exacerbated by the fact that Graeber continually talks around the idea of debt in other ways, mentioning debt crises (without reflecting deeply on why these happen), the periodic use and disuse of coinage (which apparently is just as bad as debt in terms of enabling the capitalism monster), and any other phenomenon related to debt, without weaving these observations into a coherent whole.
> In other words, I am now angry at myself for paraphrasing the book, and trying to put theses into Graeber's mouth, because this is such a rambling, confused, scattershot book that I am doing you a disservice by making it seem more coherent than it really is.
> The problem of extreme disorganization is dramatically worsened by the way that Graeber skips merrily back and forth from things he appears to know quite a lot about to things he obviously knows nothing about. One sentence he'll be talking about blood debts and "human economies" in African tribes (cool!), and the next he'll be telling us that Apple Computer was started by dropouts from IBM (false!). There are a number of glaring instances of this. The worst is not when Graeber delivers incorrect facts (who cares where Apple's founders had worked?), it's when he uncritically and blithely makes assertions that one could only accept if one has extremely strong leftist mood affiliation
have you read the book? The book is an exploration, and an interrogation, with so much to learn from that to say that about it seems pretty philistinic.
Maybe you were just summing up the review you linked from Noah Smith. I read most of it, it's a bit meh but Noah doesn't really seem to be trying too much in it. This though: "leftist mood affiliation". That's cheap 'preaching to the choir' language.
If you have a link to a more serious review I'd genuinely like to read it.
I don't think there have been any fundamental changes since 2000 that would incentivize make communication from large public corporate entities to be more honest or logically rigorous.
Input: A year and a half ago when we began scouting for this type of machine learning-savvy engineer —something we now call the machine learning Software Engineer (ML SWE) — it wasn’t something we knew much about. We looked at other companies’ equivalent roles but they weren’t exactly contextualized to Lyft’s business setting. This need motivated an entirely new role that we set up and started hiring for.
Target: We invented a position called a machine learning software engineer.
Input: First, candidates on the ML SWE loop go through Lyft’s hiring review. The review is a regularly scheduled session for a committee to study candidates with an unbiased perspective and decide whether to hire them. Working alongside the review committee is a separate panel of interviewers that provides technical feedback. This feedback is designed to help the committee decide if there’s a fit and, if so, the candidate’s technical level. At first glance, this review process may seem cumbersome. Examining the checks and balances more carefully, however, we notice that they are intentionally introduced to put friction on the hiring process. Having a consistent review committee unifies standards and eliminates bias.
Target: A committee and a group of interviewers evaluate a candidate for fit and technical level.
Input: Despite the what-ifs, being transparent about how we design interviews can improve our interviews. Call it enlightened self-interest: candidates invest time to talk to us and we mutually benefit from learning if there is a good fit. Even if there isn’t an immediate fit, positive experiences build brand and improves candidate sourcing. Maybe the candidate can reapply when the timing is better. Practically, hiring an engineer easily costs tens of thousands of dollars. By showing how we iterate on our interviews, we reveal what we truly care about and how we try to probe at them, hopefully adding to the virtuous cycle for the hiring pipeline.
Target: We don't respect the time of our readers and are hopelessly unenlightened on this fact.
Target: We need to route taxi cabs.
It’s super innovative and fit to our business context!
> We looked at other companies’ equivalent roles but they weren’t exactly contextualized to Lyft’s business setting.
Ok, so what does Lyft need then?
> What are Lyft’s challenges (and can a specific role help)? > What should the role be with respect to the organization’s goals? > What are the desired skills, knowledge, and talents given the expectations for the role?
Umm. Isn't that what every company hiring looks to address?
> What’s left is to define the necessary ingredients for what a successful hire looks like vis-a-vis (3) in Lyft’s context: > > Skills acquired through practice, > Knowledge learned through study and personal experience, and > Talents that make each candidate unique.
Ok, I'll stop reading now since "Lyft's context" is no different from any other company let alone one looking to hire a Machine Learning Engineer/Scientist.
I'm disappointed the author wasn't more specific about where the line is drawn between "ML SWE" and "Research Scientist"/"Data Scientist" when it comes to the core ML competencies like model selection, evaluation, and design.
Having worked in teams with Data Scientists and "ML Engineers" it's been murky whether me and others in my team that were not the former were Data Engineers, Backend Software Engineers, "ML SWE", or "Software Engineer - Machine Learning".
This area of software is rapidly developing and there's no semblance of the bright line that tends to exist between Backend Engineer and Frontend Engineer for ML-involved engineers.
I'm personally interested in moving into the "Software Engineer - ML" space (or is it "ML SWE"?) and thus I have to find out what the right balance between Software Eng skills and researcher skills is. I think I have a decent sense of real-world ML basics but would quickly flounder when pressed for details on mathematical technicalities of different modelling approaches.
Lyft's job listings have these items for "Software Engineer - Machine Learning":
> 5+ years (or Ph.D. with 2+ years) of industry or research experience developing ML models
This really seems like the purview of a research scientist or a data scientist, if I understand the meaning of "developing" correctly.
> Proven ability to quickly and effectively turn research ML papers into working code
This seems fair enough, but my experience has been that the data scientists are doing this mostly, while the "Software Engineer - ML" people are preparing data pipelines or batch training systems.
> Deep knowledge of ML libraries like scikit-learn, Tensorflow, PyTorch, Keras, MXNet, etc
I know it's a job ad and thus it is a 'wish list', but this seems unreasonable.
Guess I'll just keeping studying 'all the things'. Brb in 5 years.
I really want to work in this field, but it seems I must have a PhD at minimum and not sure if I want to take that step yet.
I manage a team of machine learning engineers in a mid-size ecommerce company and I can tell you that the same person who is optimizing Dockerfiles for better layer reuse & figuring out how our CI pipeline will safely get secrets needed to retrieve model files for integration tests is the same person researching new variations of triplet loss in a paper from arxiv they present in our team’s journal club & developing Bayesian hierarchical regression models and explaining why partial pooling using an industry-specialized prior distribution actually produces coefficients in the model fit that have a meaningful improvement over OLS for some business outcome.
The same people that are refactoring a collection of unit tests to be parallelizable and save us 45 seconds on every test run are also running huge hyperparameter tuning experiments to get a learning rate scheduler that allows us to reduce training time for an in-house deep GRU neural network from 24 hours to 15 hours, and they are defining infrastructure as code tooling to even create the very GPU environments where this training is taking place.
The skill set really is extremely different from general backend engineer with a proclivity for hacking around ML models and also is really different from “research developer” who rarely deals with end to end systems or necessary concerns of production or quality code factoring, and also is super different from data science which is effectively just ad hoc business analytics but with more impressive pedigree on the resume.
I’d define a machine learning engineer as taking someone with several years of graduate experience in statistics / machine learning inclusive of formal probability theory, analysis, topology, Bayesian stats (or work + research experience that is equivalent, though it’s super rare for this to be able to replace formal graduate math training), and then adding senior level skill set in high-performance computing, generalist backend engineering, system architecture, and full management of the lifecycle of complex production systems.
The only piece of common modern engineering that I would say ML engineers typically don’t have a senior-ish level of command over is frontend development (though some do out of just hobbyist interest).
I've worked with stats/comsci/bio-informatic/ML PhDs at multiple multi-billion-dollar software companies now and it's certainly not true that even the majority of people with graduate stats training are excellent software engineers. Would love to work at a company where that's true, I just don't think that's the common case at all.
> The skill set really is extremely different from general backend engineer with a proclivity for hacking around ML models and also is really different from “research developer” who rarely deals with end to end systems or necessary concerns of production or quality code factoring, and also is super different from data science which is effectively just ad hoc business analytics but with more impressive pedigree on the resume.
Totally agree. The terms are very messy though. The "ML Engineers" with PhDs in ML are still called "Data Scientists" at Zendesk, for example. At other companies "Data Scientist" just means 'Data Analyst that knows SQL'.
> I’d define a machine learning engineer...
Would be happy with this definition. It's much clearer than what you can gather from Lyft's post. It's close to the definition I currently hold to. On this definition I've enrolled in graduate Maths+Stats as I'm still almost entirely a Data Engineer / Backend Engineer in terms of skillset.
There is not a division of labor where one person makes the model and then throws it over the fence to a team that manages its production usage and lifecycle. That team over the fence would not be able to do it, and I’ve seen this attempted org structure fail hard everywhere its been tried for ML services.
What’s telling is that you bring up the personnel risk but you don’t consider the value add. When Average Corp hires someone, it’s because that person brings more value than they cost, period. It’s not because the person is “not risky” in some vacuum of decision making like you’re painting it out to be.
> Dead no from me.
That’s fine and all, but it usually indicates tech death of a company, and likely you have brain drain in more area than just machine learning. If you aren’t willing to take the risk and do what’s needed to structure the work and job to extract value from high performers, that’s a sign of corporate mediocrity and I think anyone with the skill set to be an ML engineer like this would already not even be applying to work in a place like that.
Honestly the risks are even worse than your comment says. There are also big risks around keeping this person intellectually engaged, giving them valuable job experience.
You pretty much have to pay them a lot, give them good work life balance, give them budget for conference travel & continued learning, give them meaningful upward career & compensation growth, and give them meaningful projects.
I look at this and think, yes, if the business can’t give all those things and still be coming out ahead on the person’s productivity, then you don’t want an ML engineer.
But more often the company needs an ML engineer and absolutely would gain more from their productivity than they lose on supporting all those job quality aspects — yet managers just take superficial offense at these demands and balk at the idea that you have to provide meaningful projects and career growth instead of just barking orders and expecting them to put up with work that does not help them grow.
When did this happen? I haven't interviewed for a few years, but nothing was open or clear then
My question is: why the hate? Isn't a machine learning engineer just as valid as any other engineer?
Instead, it was five algorithm quizzes... all in Java. I had an hour to finish it and, after spending the first fifteen minutes searching for Java standard library stuff just so I could write the code I wanted, I decided this was a waste of my time and closed the tab. This was all before I could even speak with someone who works at the company to find out if I cared to move forward.
I worked as a hiring manager for a couple of years and have had input into the hiring process for more than a decade. I've never found it difficult to screen someone out who had no place applying to begin with by taking a fifteen minute call. This at least gives them a chance to ask some questions and they don't feel like you're yanking them around.
Even at the ML level, engineering is more about applying known good techniques and less about new innovative ideas.
You can't fake this. Or to put it better, even if the code isn't yours and you can do this well, it doesn't even matter that the code isn't yours as in doing this you by definition have the skills and knowledge to reimplement it anyway.
Have you actually tested this out?
Regarding your other point I agree with you, but if the main point is to evaluate understanding then the difference doesn't really matter. If you're testing for ability to invent concepts etc, then this might be more important to you.
>As a candidate, it’s easy to get a foot in the door and be evaluated by an interviewer.
Do people really feel that way about a ML SWE postion at Lyft? I would be interested to see what percent of applications they receive make it to the door.