Thoughts on ML Engineering After a Year of My PhD
shreya-shankar.com
shreya-shankar.com
- "Plateform MLE" is juste regular Devops and Software Engineering. It's not because we're dealing with models and their accuracies that it is fundamentally different from what uses to be Devops before. We don't need to make a "special" title out of everything in Data Science.
- I still like to introduce "MLOps" into the conversation, thus making it special, and infringing the rule above. Oups.
- over fitting slightly on recent data, without accounting for gaps in modeling, is likely to lead to dire situations. See also : March 2020, when all forecasts went bananas. And , no, retraining at that time did not improved the situation. That's what the MLE was talking about here : > “I know it’s not really addressing the data drift problem,” ; they were right.
- Everything about data and model drift is just the tip of the iceberg: What happen when your model ends up in production ? It will start affecting the behaviour of the very thing you're trying to predict. A prime example of this on the retail markdown case : did you sold more of that article because the product was rated better (as qualified by the marketer to compensate for lack of control over markdowns), and the in-store stock was higher, or did your markdown on that item had the effect you were looking for and the rating was actually okay ? Did the sales went down this markdown season because your model had a perfect but unattractive markdown strategy last season ? This is already very difficult to model properly, let alone trying to measure their contribution to the drift. Good luck with that.
Indeed, again part of the reasons I think MLOps is a problem. One ideally needs a causal theory and rational expectations regarding how and why a model will work: you should be able to explain it. The reason is that there are many more ways for things to look like they work, then ways for them to actually work. Forcing -- what should be -- the scientific effort of modelling into a cut box, tune, re-train, monitor workflow is reductionist and will invariably make it near impossible to exercise proper due diligence IMO.
Scientific discovery is generally not a sausage machine -- its messy and dependent on what happens in brains more than anything else. I think its a fools errand to expect differently from -- what should be -- the same abstract tool box applied to commerce.
MLOps is as much as an issue to Data Science than DevOps is to Software Engineering. Automating things doesn't make them stupid or constrained ; I don't know why you would think this creates a "cut box" workflow and sabotage the beautiful Scientific process you imagine is happening. Maybe you have an issue with some Data Scientists you are working with ?
To your other point below :
> The DS cuts a black box which MLOps tunes, monitors and re-trains.
MLOps is not about adapting or tuning a model. Not even close, it's just completely out of scope. MLOps is just a set of best practices to ensure you can compare and reproduce previous outputs (a model, plot, shap output, png, or json benchmark report, whatever you want) at any given point in time. What you do with that model, and how/when you want to train it, is your business alone. We need to call these best practices "MLOps" because it's a significant enough break from regular software engineering practices : suddenly pinning a git tag is not enough to ensure reproducibility, now, we also need to pin datasets to their specific state in time. And what about outputs that may improve over as a result ? Same.
Calling them "ML" Ops is actually a bit disingenuous. Optimisation cases (with constraint or MIP solvers) that have nothing to do with ML and models also benefit from the same framework.
> It puzzles me when people say that small companies can’t retrain every day because they don’t have FAANG-style budgets. It costs a few dollars, at best, to retrain many xgboost or scikit-learn models.
if re-training every-day actually gives you a significant business material benefit (excepting cases where you specifically want to significantly over-weight recent data - phone keyboard autocomplete predictions as an example), you likely don't have a model that is actually picking signal from noise or generalizing on the data. This is memorization, not learning. This is essentially how you get expensive ML disasters.
I'm not convinced it actually pays, though.
Generalizing individual experiences to general prognostications sells well as a genre, especially in software. This writing pattern was epitomized by OOP thought leaders in the 90s/00s, but was also present in the process evangelists who followed and the more iconoclastic folks like Dijkstra who preceded. However, actually following such general advice without taking into account the author's context can easily end in tears.
Ideally, these sorts of pieces should be written up as case studies rather than as general prognostications. With the context that a case study provides, these pieces can teach us a lot about how to engineer systems. Without that context, the reader is left to blindly follow advice that may not be relevant to their context. Or, best case, reverse engineer the context.
The fact that refitting once a day improves real world performance actually makes me think that the problem/data they work on is highly non-stationary, not that the model is memorizing. If it was purely memorizing then the model would perform poorly on all non-training data, and would not work for even one day.
To your point about large datasets - the large datasets are what allow learning to take place. With the most common forms of models we have now, they will memorize when they only have a few examples, and only “learn” when the training data is large enough. There is work to improve learning from a handful of examples, but in many of these cases they require a model that was already trained on the domain in question, and then are specialized to a specific use-case.
GPT-2 was for a time the best lossless text compressor (lossless compression is just memorization) - https://bellard.org/libnc/gpt2tc.html
Ideally, we would simply never stop training.
From my experience managing data science teams (small teams - max 20 people), I always preferred for my Task MLE team mates to also do the Platform MLE work. I would not hire someone just to be a Platform MLE because they would be too distant from the day-to-day Task MLE needs.
I like the way that Google SREs think about this - there's toil (Task MLE parts of the job) and then there's automation (Platform MLE parts of the job). Every programmer on the team should have toil and should be given enough time and freedom to address their most painful toil through automation.
Distinguishing between Task MLEs and Platform MLEs so strictly is dangerous for anyone that applies this dichotomy in practice.
I guess the article author never explicitly stated that they have to be different people, but I got the sense from reading the article that this was an unstated assumption on their part.
So true. This is a pretty easy flag that someone hasn’t actually been down and dirty with trying to train any actual models and doesn’t know what they’re talking about
Re: monitoring -- When I was doing automated trading, we of course had automated alerts to stop systems that were outside expected parameters. However, as OP describes, there was also a long tail of degradations where there was no great precision/recall tradeoff on flagging "real" issues. Instead, we put effort into "calibration reports" that visually surfaced as much information about training and performance as our eyes could handle. These would include things like over time plots and PCA plots for recent feature histories, where it was more an opportunity to spot patterns than an explicit metric. Reviewing these for 15m each day with our own eyes was much more effective at detecting a long tail of unanticipated degradations than we would have been at anticipating and coding for each one explicitly -- and it help build intuition that fueled new research ideas.
> I have done enough research on production ML now to know that it pays to simply overfit to the most recent data and constantly retrain. Successful companies do this.
We all know that overfitting is bad (e.g. in time-series forecasting, the past isn't always representative of the future). Depending on your domain, more recent data may be more valuable than older data, sure. The solution is not to overfit to recent data!
In my experience, it is to design features which take into account recency. For example, in a particular quantity we wanted to forecast, we found out that using ~7 days worth of data was better than using multiple months, due to the data being non-stationary (the mean of the quantity was changing over time). What we did was combine features with an exponential decay with the appropriate decay constant, to great results.
I laughed out loud. The rest of the essay is so on point I'm attributing the author full marks for a classic example of the dark, laconic understatement that builds so many bridges amongst technology professionals.
Man searching for his keys under a street lamp in the dark. Police man asks “Is this where you dropped your keys?”. Man says “No, but this is where the light is”.
Maybe you're conflating this with something else ?
This sentence seems to contradict itself.
Otherwise, I'm familiar with MLOps :-) Wikipedia describes it as the intersection of ML, devops and data engineering, which is also how I've experienced it. What data is available (already "plugged in") paints a path of least resistance for a data scientist (DS) who then goes on to look for solutions within that data (see joke above) -- not necessarily, but it tends to go like that. The DS cuts a black box which MLOps tunes, monitors and re-trains: more path of least resistance. This ends up reflecting in hiring decisions, and so on.
Again, not necessarily the case, just a dominant pattern I see. I suspect part of the reason is the movement of devs into MLEng and MLOps and analysts/devs into DS because analysts/devs typically do not have research backgrounds. It causes DS -- which initially was almost exclusively PhDs -- to be less about describing phenomena using maths and more about fitting black boxes. Further, "ML" as taught in Google courses (centered on Tensorflow) targets devs and literally paints the process of modelling as features, topologies and parameters. There is a selection pressure in that direction -- presumably due to labour supply, IMO to the detriment of the craft, but hey-ho.
> I have done enough research on production ML now to know that it pays to simply overfit to the most recent data and constantly retrain. Successful companies do this.
I don’t understand why that is
> Sometimes, I was so scientifically sound that the business lost money. I automated a hyperparameter tuning procedure that split training and validation sets into many folds based on time and picked hyperparameters that averaged best performance across all the sets. I only realized how silly this was in hindsight. I should have taken the hyperparameters that yielded the best model for the latest evaluation set.
What's silly here is thinking that the minor adjustment of hyper parameters from set to set is likely to make a difference. This might hold for some niche deep learning problems but it sounds later on like she isn't doing this. I rarely see an optuna parameter optimization affect the AUC of a model by more than 0.02 vs arbitrary choices. Most business problems tend to be pretty simple. I just can't imagine this parameter choice makes a difference.
> I have done enough research on production ML now to know that it pays to simply overfit to the most recent data and constantly retrain. Successful companies do this.
Nonsense. Dangerous nonsense.
> It puzzles me when people say that small companies can’t retrain every day because they don’t have FAANG-style budgets. It costs a few dollars, at best, to retrain many xgboost or scikit-learn models. Most models are not large language models. I learned this circuitously, while doing research in ML monitoring.
This is accurate. I find it weird that people feel the need to retrain models so frequently though. Context matters I guess. But then they talk about data drift...
> Anecdotes like this really get me in a tizzy. I think it’s because what I thought were important and interesting problems are now, sadly, only interesting. Researchers think distribution shift is very important, but model performance problems that stem from natural distribution shift suddenly vanish with retraining.
It sounds like this person is just retraining their models at a high frequency until a hyper parameter search randomly produces something that looks good on an evaluation set. I don't see any evidence of them ACTUALLY EVALUATING THEIR PERFORMANCE. The missing piece to all of this is taking the predictions they made with this process and seeing how it compared to the unseen future outcomes in retrospect. If you think you're fine because you make a new model every day that works for yesterday's data, which they openly imply is overfit to that period, you might just be making a new mistake every day. A good AUC or R2 on your evaluation set doesn't guarantee that it'll be good on new data.
> I had a hard time finding the gold nugget in data drift, the practical problem. Now it seems obvious: data and engineering issues—instances of sudden drifts—trigger model performance drops. Maybe yesterday’s job to generate content-related features failed, so we were stuck with old content-related features. Maybe a source of data was corrupted, so a bunch of features became null-valued.
What? This seems pretty crazy to me. You should have alarm bells built in that freak out if important features are turning up null or corrupted. This person seems to have no idea if their model is working! They just know it worked for yesterday. They allude to doing this later on, but even then... broken data pipelines is not what people typically mean by data drift.
I see no discussion of real performance evaluation, understanding how the models work, or how to make real use of the model outputs.
'Data drift' sounds like some BS one tells their manager when they don't understand what's really happening.
> I have done enough research on production ML now to know that it pays to simply overfit to the most recent data and constantly retrain. Successful companies do this.
As long as they are not AB tested statement likes these are worthless. I recently tried updating a model daily vs weekly and it made aboslutely no difference so why would I care ?
Anyway, thanks for your comment I know I'm not alone.
The overfitting phenomenon I've seen at the companies I work/have worked with are all doing some kind of ranking. None use auc to evaluate---in fact, I don't see many production models evaluated with auc because usually some threshold is fixed before deployment, or the client has their own threshold they will act on.
Re: hparamaters varying the metric---a point on the ROC can change a lot while the auc stays relatively the same, especially in class imbalance settings.
Re data validation---at large companies with 10s-100s of models with hundreds of thousands of features, often data gets corrupted. Alarm bells do go off but there is alarm fatigue. Furthermore, not all Task MLEs are Platform MLEs and vice versa...a lot of people have to build tools to monitor ML pipelines they don't own.
Anyways I really appreciate your comment & it made me realize how ML people have vastly different experiences. I will clarify assumptions on the task & evaluation next time I write :)
> usually some threshold is fixed before deployment, or the client has their own threshold they will act on.
Why would anyone do this? I mean I don't care too much about which metrics they choose to use but this seems like a really weird choice to me as its blind to any of the typical confusion matrix outcomes you see in the model validation.
I am very curious to know if you track model performance on new data made in retrospect. The frequent retraining is foreign to me. I get that its not hard to do but I would feel really uncomfortable if my models changed in a meaningful way from day to day. Maybe it's just because my models tend to deal with the outcomes of natural physical processes
Once the model makes predictions, they need to be surfaced to a client/user. For example, I once worked on ML pipelines for car companies that predicted which parts of vehicles would fail & when. Every day, we generated probability of part failure for each part & vehicle. First we decided to give the car companies the probability of failure for each part, but they didn't know how to act on that (like does 60% engine failure in the next month mean I should get the engine replaced or not?). At the end of the day, what was evaluated was: how much $ the car company saved in performing a car part recall, which was a function of the binary decision whether a client replaced the car part or not. So this is what I mean by choosing a threshold before deployment (i.e., tell the car company all the cars & parts that should be fixed, > 50% or some threshold failure) or leave the thresholding to the client (i.e., give them the probabilities and see what they do).
> I am very curious to know if you track model performance on new data made in retrospect. The frequent retraining is foreign to me. I get that its not hard to do but I would feel really uncomfortable if my models changed in a meaningful way from day to day. Maybe it's just because my models tend to deal with the outcomes of natural physical processes
Yeah, a lot of ads, content recommendation, personalization, fraud detection, etc tasks have frequent retraining. It feels uncomfortable for sure. When I was an MLE I would do all sorts of backtesting, but it felt kind of like voodoo magic because even if I validated a change to the model on old data, who is to say the change will positively impact future predictions (arguably all that matters to the business)?
My point is that you are the person who can do this. Not up front, but you have the ability to look at your past prediction of future outcomes and see if what you're doing is actually working or not. You already had the predictions. You now know the actual outcomes.
How does the performance metrics of your model's predictions from back then on your validation set compare to the performance of your models when they were used for real? How would the prior day's model do. Or the prior year's?
Often your data validation process does things like running descriptive statistics (means, standard deviations, etc) on particular columns and making sure they are within reasonable ranges.
Strongly typed languages don't have any particular advantage for this kind of thing.