mmq

Hi @the_duke,

disclaimer I am one of the authors of an open-source solution (https://github.com/polyaxon/polyaxon) that specializes in the experimentation and automation phase of the data-science lifecycle.

Our tool provides exactly the kind of abstraction you mentioned:

* Training, data operations, and interactive workspaces (https://polyaxon.com/docs/experimentation/)

* A scalable history and comparison table (https://polyaxon.com/docs/management/runs-dashboard/comparis...)

* Currently pipelines and concurrency management is on the commercial version (https://polyaxon.com/docs/automation/) but several companies use Polyaxon with other tools like Kubeflow (https://medium.com/mercari-engineering/continuous-delivery-a...) or it can be used with MetaFlow for the pipelines part.

I would really like to hear your thoughts and feedback.

There are several other platforms/tools taking different approaches to productionizing ML workload. At Polyaxon[1], we used to create a container for each task and also log the git commit for reference, and provide ways to push the container to an in-cluster registry or a cloud registry. In the new version of our platform, we basically improved the process by injecting specific git commit inside pre-built docker images to not only reduce the build time, but also to allow easier integration with external tool such as Github actions.

[1]: https://github.com/polyaxon/polyaxon

Argo implements its own scheduler AFAIK, otherwise how would it manage dependencies and the execution graph. The part that Argo is using K8S for is orchestration, which Airflow can do as well with the KubernetesPodOperator, but it's not a cloud native solution and it spins the whole scheduler and backend for each task.

We tried to make Polyaxon[0] work with Airflow for Machine Learning specific workflows, but it was very painful and it does not have a good state/artifacts management, which leaves the users tweaking around. We end up making a simple abstraction on top K8S, much easier, to provide features for parallel executions, dependencies, failure handling, retries, ... as well as handling ML specific graphs such as hyperparameter tuning and distributed scheduling.

[0]: https://github.com/polyaxon/polyaxon

I made one of those posts and it got flagged almost immediately, and I honestly just wanted to know if the issue is on my end or not, GitHub's status page shows that all services are up.

> if you put an engineering team on it that know how to write automated release procedures

I think surfacing the data is just the first step, often times data scientists need to run some data exploration, the process is generally iterative, and so they need to run several experiments, resume or restart some experiments, scale training with distributed learning using several machines, or run hyper-parameters tuning, which means handling failures, visualize and debug results, before deciding if they should deploy a model. Once a model is deployed the story does not end there, because models become stale and need to be retrained. There are other issues related to compliance that need to be handled as well, and many other problems related to governance, a/b testing, ...

The good news is that there are several open source initiatives to solve several of these problems, at Polyaxon [0] we are trying to solve some of the aspects related to the experimentation phase.

[0] https://github.com/polyaxon/polyaxon

Polyaxon is an open source machine learning automation platform. It allows to schedule notebooks, tensorboards, and container workloads for training ML and DL. It also has native integration with Kubeflow's operators for distributed training.

https://github.com/polyaxon/polyaxon

Anyone with more info about the difference between TorchElastic and PytorchOperator?

Working on Polyaxon (https://github.com/polyaxon/polyaxon), an open-source machine learning experimentation and automation tool.

Quite interesting to hear this, it's very much the same observations while working with some customers, Airflow is a very mature and an amazing tool, but it does not have a good state/artifacts management, which leaves the users tweaking around, scheduling is centralised, and is not designed for ML workflows, i.e. hyperparams tuning, distributed runs, ... the kubernetes support is also quite limited.

Polyaxon[0] took a similar approach to FLyte, i.e. for authoring specifications: strongly typed system in protobuf + intuitive yaml specification + sdks in Python/golang/java/... It also treats operations (tasks in Flyte) as first class citizens and allows to run them in a serverless way. Users can choose to register repetitive operations as components and share them with a description and a typed inputs/outputs.

[0] https://github.com/polyaxon/polyaxon

Since it's possible to create custom components, one can for instance create a component for packaging models, for example the component can extract the model from a path (mounted volume or blob storage), create a python package with a requirements file, and a microservice based on flask, everytime a user wants to promote a model to production, she can run this component, or she can also add it to a workflow to be triggered after a training or hyperparams operation.

From Polyaxon's side, we will be providing a set of reusable components, some of these components will be targeting deployment and packaging, for instance aws lambda, azure ml, sagemaker, open source projects for deployment (some of them are mentioned in this HN thread) Users can also contribute components as well or create them to be used inside there organization.

For versioning, all component have versions, and all runs have full overview of their dependencies and provenance (inputs/outputs). This gives all information about when and how a model was created.

The platform knows how to create and manage services, e.g. tensorboards, notebooks, dash, simulators for RL agents ..., so one can also deploy the model as an internal Polyaxon service to be used as an internal tool or for testing purposes, the only thing to keep in mind is that the API endpoint will be subject to the same access rights as other components create by a given user.

I will try to be as objective as possible answering this question, since I am working on a project in a competing space.

Argo workflow is a pipeline engine that is cloud and kubernetes native. It tries to solve graph and multi-steps workflows using containers on Kubernetes, It can be leveraged for ML pipelines as well as other use-cases.

Kubeflow is a large project that has several components: training operators, serving (based on Istio and Knative), metadata (used by tensorflow TFX), pipelines, ... and integrates with other projects. Kubeflow pipelines is using Argo workflow as a workflow engine, although I think there are efforts to support other projects such as Tekton which is also a google project, and possibly TFX as a DSL for authoring pipelines in python.

The main focus for MLFlow, I think, is tracking ML models and providing an intuitive interface to model deployment and governance. The main strength of MLFlow is that it's easy to install and use.

Polyaxon has been used mainly for fast developement and experimentation, it has a tracking interface and several integrations for dashboarding, notebooks, and distributed learning. Polyaxon also has native support for some Kubeflow components, e.g. TFJob, Pytorch job, MPIJob for distributed learning.

The upcoming Polyaxon release will be providing a larger set of intergrations for dashboards, in addition to tensorboards, notebooks and jupyter labs, users will be able to start and share zeppelin notebooks, voila, plotly dash, shiny, and any custom stateless service that can consume the outputs of another operation.

The new workflow interface focuses mainly on an easy declrative way to handle DataOps and MLOps, the main idea is to provide a very simple interface for the user to go from a data transformation to training models. Since the component abstraction is based on containers, it can be used to do other operations, e.g. packaging models and preparing them to be served on other open source projects, cloud providers, or lambda functions. Also support for some frameworks such as dask, spark and flink operators could be used as a step in a workflow, ...

For hyperparams tuning, Currently, the platform has grid search, random search, hyperband, and bayesian optimization, one of the major changes in the next release is a new interface for people to create their own algorithms and a mapping interface to traverse a space search provided by the user or based on the output of another operation.

Glad to see that you are interested by using Polyaxon[0] for your MLOps. Although I was going to write a blog post about the upcoming v1.0 release of Polyaxon, I just wanted to point out that there will be a native support for different type of workflows, currently it supports parallelism and distributed learning, and in the next release there will be native support for DAGs as well. Here's a test fixture[1] of what a dag workflow will look like in Polyaxon.

Happy to answer any question or provide more information.

[0]: https://github.com/polyaxon/polyaxon

[1]: https://github.com/polyaxon/polyaxon/blob/master/cli/tests/f...

The changes are not only about making the website/app stable, it's mostly about shipping new features and product ideas.

The space where Instagram operates is very competitive, so they need to keep innovating and exploring new ideas to grow the product, increase usage, and improve retention...

Not sure why you did not have any "Aha" moment, bu the visual explanation used for the principle of superposition and approximation of functions by Fourier series is very informative. I also liked the explanation about the proportional relation between temperature and space in the heat equation. I think in most higher educational systems, they only approach this type of problems with pure theoretical solutions/explanations.

Hi @minimaxir, I just checked the readme of your project. It looks great, there's a mention to integrate Polyaxon[0] for running jobs in distributed way on Kubernetes. Let me know if you need any help with that.

[1]: https://github.com/polyaxon/polyaxon/

I might be wrong, and I also highly value and use the tools that you mentioned. However, I have to say that I see the emergence of these tools as a positive thing for 2 reasons.

The first one I think is that more and more people are now using/trying to use machine learning models in production and they discover that the workflows and tools they used to use and work with are not suited for delivering machine learning models in fast, repeatable, and simple way.

The second reason is that I objectively think that a machine learning pipeline or CI/CD system is a bit different than the one used for pure software engineering practices, partly because machine learning does not only involve code, but more layers of complexity: data, artifacts, configuration, resources... All these layers can impact the reproducibility of a "successful build". Hence, a lot of engineering is required to both ensure that teams can achieve both reproducible and reliable results, and increase their productivity.

> "I don't really see what there is to be gained by making scheduling part of the remit of pipelines."

I think it depends on the use case, sometimes the components of the pipeline aren't necessarily running on the same machine, and they don't know where and how to get access to data and artifacts generated by previous steps, and so scheduling and orchestration becomes an important component of the pipeline itself.

> "I'd rather assemble together a set of tightly scoped "UNIX philosophy" libraries and tools rather than try and use an all encompassing framework and be straitjacketed by its imposed structure."

I think the idea behind building such frameworks is to help people avoid going through the same steps of building such tool internally by "assembling together a set of tightly scoped "UNIX philosophy" libraries". In general these frameworks are using libraries and tools, and exposing an easy way to leverage them instead of spending time doing that over and over.

UPDATE: Looking at the docs, there's an example[1] using this library with scikit-learn.

[1]: https://nni.readthedocs.io/en/latest/sklearn_examples.html

I think it should probably support scikit as well as any other library, since it's only making suggestions of hyper-parameters based on recorded/historical observations or random evaluations.

At least that's the behaviour of the platform[1] I am working on.

[1]: https://github.com/polyaxon/polyaxon#hyperparameters-tuning

I have attempted to do something similar in 2017 [1]. A couple of issues I noticed:

* The field is evolving quite rapidly, and so most options will require a lot of configuration, which IMO is not suitable for a declarative approach.

* It's hard to debug, eventually you will need to dive into the code at some point.

* Extending the library would mean touching the code anyways.

One major difference here, is that Ludwig tries to expose one interface, that for instance users without python knowledge can use, and inside a company, some machine learning engineers can extend the tool and do support by debugging other users use-cases.

I think at some point such approach can be useful for a very limited mature use-cases.

[1]: https://github.com/polyaxon/polyaxon-lib/blob/master/example...

It actually gives a lot of idea and validation for concepts that I was thinking about for my project https://github.com/polyaxon/polyaxon, namely for going from trained model to serving those models on other platforms in a seamless way. I also think that high-level articles like this one are really good for people who are trying to build internal tools around ML/AI ops, people can get more or less ideas about what and how other companies are managing their ML pipelines and how they impact the productivity of their teams.

I made this code https://github.com/mouradmourafiq/tensorflow-lstm-regression at least 6 months earlier than the first push in the github repo you mentioned, and I am sure many people did some version before that. A lot of people sometimes just need a reason to play with some technology. I assume that OP used stock market to learn about LSTMs.

Can you elaborate a bit more about your usage of GitLab CI/CD for model management/development. I am currently working on a platform [1] that tries to solve some of the issues mentioned in the article, i.e. improving data scientists' productivity and velocity, compare models, solve reproducibility issues...

[1] https://github.com/polyaxon/polyaxon

Polyaxon, https://github.com/polyaxon/polyaxon, is an open source platform that tries to simplify not only running notebooks on kubernetes, but also tries to solve issues related to scaling, tracking, and reproducibility.

Disclaimer, I work on Polyaxon.

Very nice article.

I always wanted to extend my tutorial[1][2][3] with a 4th part, to make the connection between Euler Gamma function and Riemann Zeta Function.

I will link to your article, it's really well written.

[1] https://mourafiq.com/2015/08/30/extensions-of-the-factorial-...

[2] https://mourafiq.com/2015/09/09/extensions-of-the-factorial-...

[3] https://mourafiq.com/2015/10/09/extensions-of-the-factorial-...

I can't say much about long-term roadmap apart from the fact that the platform will be open source and that it will try to introduce features that will increase the productivity of data scientists.

For short-term roadmap, I am trying to work on stability, it's very hard to have default values since you don't know how it will be used, e.g. on Minikube or by a team scheduling a lot of parallel experiments, so what I am trying to do is at least having an automatic or simple way to scale workers responsible for scheduling, hyper params tuning, and monitoring.

For tuning, the platform will keep supporting some algorithms to automate the hyper params search, maybe introducing more priors for the Bayesian optimization, I also think more tests are needed to validate the behavior of the Bayesian optimization and Hyperband.

For visualization, currently, you can start a Tensorboard for any project created on the platform, but there are some problems with this assumption, if the project has a lot of experiments, Tensorboard becomes slow to irresponsive. Next release will introduce the possibility to create Tensorboard jobs per experiment or per hyper tuning experiment group, and possibly any collection of experiments to compare them.

The platform collectes already metrics from experiments, so a basic visualization is also planned to have a quick overview before diving into a Tensorboard.

And most importantly, I think there are some usability issues that need to be solved to make the experience better.

There are also a couple of ideas around team collaboration that will be introduced in the mid term.

I think the target audience, is individuals or small teams who want to have an organized workflow, immutable and reproducible experiments with an organized and easy way to access logs and outputs.

The platform also provides a lot of automation to schedule concurrent experiments.

There are a couple of things that need to be polished to be used, notes on experiments, notification of finished experiments, especially if you are running hundreds of experiments.

Depending on how organized you are, many times you will end up with experiments that you did not know how you started, having a platform that takes care of that could be beneficial.

If you have already a cluster, for running your experiments you will most probably end up ssh-ing to the machine to check which experiments finished, probably in a screen, and their results and logs, Polyaxon simplifies that part as well.

As you said if the library/framework provides deterministic computation, then it should not be a problem, I am not sure if it's already fixed in Tensorflow, but parallel computation and order of operations could all influence the reproducibility of the run, so sometimes you need to provide that in your code.

What Polyaxon provides, is a way to restart a run based on as many parameters collected, by default it will use same code based on the internal git commit, it will reuse the same configuration, same Dockerfile, and if provided, it will use same resources, CPU, GPU, and memory of the original run, if the experiment had a node selector, the restarted experiment will be as well scheduled on the same node.

I just want to add to the other comments that Polyaxon focuses on different aspects of machine learning reproducibility than Pachyderm, although Polyaxon will be providing a very simple pipelining abstraction to start experiments based on previous jobs, triggers, or schedules, or provide the possibility to run post-experiment jobs. It will not focus on data provenance the same way Pachyderm does. In fact, Polyaxon and Pachyderm could be used together.

As @syllogism said, you can use any framework/library to train your models.

A couple of libraries have a special integration, Tensorflow, MXNet, Horovod, and Pytorch, they have a section in the specification for users who want to run distributed experiments.