Hacker News Reader

Comment by simonw | Hacker News Reader

I see this as more of an organizational challenge than a technology challenge.

Getting ML models into production isn't particularly hard... if you put an engineering team on it that know how to write automated release procedures, design architecture that can scale and build robust APIs to surface the data.

But in many companies the engineers with those operations-level skills and the researchers who work on machine learning live completely separate lives. And then the researchers are expected to deploy and scale their models to production!

That's not to say that this organizational problem cannot be solved with technology/entrepreneurship. If a company can afford it it's likely much cheaper to pay an external company to solve your "ML in production" problems than to re-design your organization such that you equip your internal ML teams with the skills they need to go to prod.

View on HN

I agree that a lot of the challenges around production ML are organizational, but I think in many companies, it has more to do with a lack of engineering resources than it does the separation of eng and data science (though that certainly happens).

Building and maintaining ML infrastructure from scratch is a big project. That's why you see FAANG companies hiring for ML infrastructure/platform engineers. Most startups don't have the extra cycles for that big of an undertaking, and so you see a lot of slapped-together, hacky solutions to putting models into production.

I'm biased in that I work on Cortex ( https://github.com/cortexlabs/cortex ), but I think that open source, modular tooling that removes the need to reinvent the wheel is going to have a big impact in terms of making production ML more accessible.

I disagree. It’s not about getting the data where it needs to be. It’s about data version control at a very fine level with very large datasets (in a way that is efficient). It’s about detecting changes in model results base don changes in data. It’s about tracking provenance of data in the datasets. It’s about potentially controlled access to the data (eg allowing models to use health care data without actually knowing the underlying data). It’s about detecting bias in datasets over time.

It’s actually quite complex, which is why generally speaking very few people do anything like this. I am unaware of any general solution to this problem, either in industry or academia.

You've brought up a lot of very interesting points that we're actually looking to solve regarding data distribution changes, data version control and reproducibility, privacy guards and bias detections with our startup Orchestra (https://orchestrahq.com).

Would love to chat if you have further thoughts around the subject - there's a ton of problems we're looking to tackle in the space and would be good to get input.

(Tecton CTO here) You’re absolutely right that ML projects can’t be solved with technology alone. Besides the right tooling, they also require process, organizational setup, buy-in from multiple stakeholders, etc. By itself, no technology will turn a company into an “ML-first” company. Both technology and organizational problems need to be solved.

A while back, we published a blog post that discusses how we approached these organizational challenges at Uber: https://eng.uber.com/scaling-michelangelo/. With Michelangelo, we found that the right tooling can both solve technical challenges and help with some organizational challenges. For example: If a standardized and centralized platform is the path of least resistance to get ML into production and solve your business problem, you get the organizational benefits of that centralization (governance/visibility/collaboration) along the way.

> if you put an engineering team on it that know how to write automated release procedures

I think surfacing the data is just the first step, often times data scientists need to run some data exploration, the process is generally iterative, and so they need to run several experiments, resume or restart some experiments, scale training with distributed learning using several machines, or run hyper-parameters tuning, which means handling failures, visualize and debug results, before deciding if they should deploy a model. Once a model is deployed the story does not end there, because models become stale and need to be retrained. There are other issues related to compliance that need to be handled as well, and many other problems related to governance, a/b testing, ...

The good news is that there are several open source initiatives to solve several of these problems, at Polyaxon [0] we are trying to solve some of the aspects related to the experimentation phase.

[0] https://github.com/polyaxon/polyaxon

Reply on news.ycombinator.com