Building Thousands of Reproducible ML Models with pipe

For example, data science work at companies like LinkedIn or Facebook can end up requiring concentrated focus on model performance as they have highly developed ML capabilities.

Data scientists at smaller or less data-driven companies end up running ad-hoc queries for marketing or product teams. We are somewhere in-between at Automattic and we find value in sharing our day-to-day work so other data scientists and companies know what to expect when they hire or get hired as someone to do data science work - which is now such a broad definition that it barely means anything.

I also wanted to share our learnings about going from ad-hoc queries, one-off models and solutions to general frameworks, and make a case for custom ML pipelines by showing how this kind of work is so closely coupled with internal data.

It really isn't very magical, though. It is mainly data and software engineering work as mentioned in the post and we are still in the early stages.

I think there is some magic in the applications of ML models to actual business questions which is also a topic that I want to post about on the same blog.

It also wasn't meant as a way to recruit people, as far as I know, we are not currently explicitly looking to hire more data scientists (but also, of course, always welcome applications). I posted this to HN because people ask many questions about what exactly it is that data scientists do in the industry!

Yes, this is why we use regression, soft-margin SVM, etc. instead of hard-margin SVM. Because perfect linear separation is unrealistic.