How to put machine learning models into production

Yeah, there was a multiple dev long effort to build "complete" solution but I did what you described to satisfy impatient stakeholders. Since then the fancy solution is incomplete and mothballed while the simple deployment has been running ever since.

The next value add was consolidating data in a db with web ui so relevant non-devs can view and help add to it + easily integrate different data sources with automatic validation. I wish their was a nice open source thing here you could spin up without much effort. There's middle ground between complex/paid full solution that can scale to huge data sets and integrate with turking etc and emailing csvs or sharing google sheets links around.

Agreed! It seems like there's a lot of power in having "one to beat". As in, let's get some model up before we worry about one that needs daily updates.

You only get 80% of the gains if your data distribution is stationary and/or you can construct a non-ML solution with near 80% of the ML one. If a non-ML solution is that close to non-ML then I would just stick with that - the advantages in terms of predictability and ease of maintenance would likely outweigh small performance gains.

A model that can't be continuously trained inevitably rots due to data, interface or environment changes (and the code is typically very difficult to maintain across team members - if the author leaves it's often a ticking time bomb). If you're OK with the model rotting then it wasn't that important to your business to begin with. This is not true for all businesses.

Thanks for the share. Will check it out.

Registered! BTW, awesome panel title.

There seems to be the idea that training an ML model is like compiling code - but every "compile" leaks information into the training pipeline. Repeated testing and choosing (unless it is on a fresh draw) is an optimization step, you are optimizing on the test set.

Using a fresh draw is difficult and expensive, especially since the labels may not be available. Using A/B is expensive, multi-armed bandits are more efficient, but again there is an optimisation element there (waits for shouting to start)

Additionally surely there is a really significant qualitative judgement step about any model that is going to be used to make real world decisions?

This is almost always a very bad idea because the if/then condition you describe is business logic - as in, how do you recognize the business situation when you want to decline a prediction, and how does that change?

It is very complex because most of the time there is no simple rule such as a threshold on the confidence score of the prediction. In practice it might be more like, “if the user has more than 7 items in their cart and if the user is not a returning customer that filled out personal data and the value of their cart is greater than $100 and they have not put a new item in the cart for 2 minutes, and the confidence score of the predictor is less than 0.4, THEN don’t show the next recommended item, just display a checkout link.”

And the number of items, the cart value limit, the time since last item-add, etc., will all be hotly debated by product management and changed 5 times every quarter.

In grad school there was a professor who said of machine learning that “parameters are the death of an algorithm” - so you want to avoid coupling extra business logic parameters tightly with the use of machine learning models.

The if-then predictor is called an expert system and it is a kind of model. Combining multiple models is called ensemble model. Here, this is how the idea can sound smart, not like cheating. ;-)

On a more serious note - as always, it depends. On the logic, on the company processes and skills, how frequently will the logic change, visibility into the decision making, ... It may be a good idea or you may bring in a lot of manual work.

> To me, it talks about everything but actually deploying a machine learning model in production.

This seems to be a common theme of a lot of articles about 'how to put ML models in to production'.

kubeflow is pretty horrendously bad unfortunately. Most of the installation docs are incomplete and inaccurate, and since the workflow requires building a separate container for each submitted task (instead of separately specifying version control commit) you cannot actually get reproducible results. You’d have to scrape the state of the code out of the identified container tied to a job, since the circumstances under which the container was created for the job can be any arbitrary, out of band changes a developer was working on, such as from a branch they never pushed.

This workflow also doesn’t work well in hybrid on-prem + cloud environments because, for example, your model training might run in a cloud Spark task, but your CI pipeline (responsible for building and publishing a container to an on-prem container repo) might run on-prem. kubeflow, for example, has a hard requirement to put containers into cloud container registries, and makes assumptions about what the networking situation is allowing connection between on-prem and cloud container resources.

I think industry shifting focus to kubeflow is actually a giant mistake.