Most people doing machine learning at the moment are using a pretty bad workflow. It's difficult to avoid the trap people refer to as "grad student descent": endless tinkering, where you run two or three jobs, monitor the results, and then kick off another one. You don't really have a hypothesis in this cycle, so you don't know when to stop. At the end of the process you've generally gained intuition and insight, but nothing you can reliably pass on.
The solution to this trap is to commit to a matrix of results you're going to collect, program up the experiments, and let them run. Once you have the proper comparison, you can then decide what to do next.
Most university research groups get a grant to buy some hardware, create a cluster, and then schedule jobs on the machines using SLURM or HTCondor. These technologies are mature, but they leave individual researchers with a lot to do. You can schedule your jobs, but you have to write the experiment management yourself. My hunch is there are maybe 5-50 companies in the world with internal systems significantly more sophisticated than this.
Polyaxon brings the shiny new "cloud native" workflow to the problem. It runs on Kubernetes, which is much easier to use, especially with heterogenous hardware. I think just switching to Kubernetes and containers would be helpful to a lot of teams. On top of the cluster solution, Polyaxon brings a nice experiment management layer, with hyper-parameter search. It also manages the containerisation for you, so that the researcher doesn't need to interact with Docker directly.
There are still a number of things that are under-developed. The most noticeable are dataset management and artifact export. You currently have to do this yourself, e.g. by adding persistent disks to the cluster. I use GCSFuse to mount GCS (Google S3) buckets as directories, which works pretty well in the meantime. There are also a few defaults that could use refinement. If large clusters are being created, the management services are currently a bit under-resourced. Finally, there are a few more minor rough spots. For instance, the web app is currently a little unpolished. It's a Django app, so everything takes two or three more clicks and refreshes than you'd ideally want. A more AJAXy front-end would be nice.
There are several commercial competitors. There's a pretty obvious analogy between the use-case here and the CRM. Companies hope to own the "system of record", and be the shared space where ML teams collaborate. DominoDataLab.com , Neptune.ml , Cloudera.com , Datascience.com and others all have slightly different takes on this problem, to different degrees. Many of the above are built around a Jupyter Notebooks-based experience, and are targeted more towards workflows where the primary outputs are insights and reports rather than product development.
I think an open-source framework is valuable for a few reasons. We should be reluctant to buy in to commercial platforms for precisely the reasons vendors are so interested in this space. Lock-in can hurt here, and if the computation is scheduled via the vendor, they get the chance to tax you a % of your compute spending. Given how expensive GPU experiments can be, that's a big ongoing cost to sign up for.
I also think uploading all your training data to someone else's system is a bad idea, that your data sharing agreements often won't permit.
Finally, it's nice to have a local Kubernetes cluster for other reasons. Kubernetes is basically an OS. Polyaxon is an app that runs on that OS. This is nice: you can develop other apps to work with it as well. In contrast, if you rent a service, the easiest way to meet your next requirements will be with further services. As soon as you hit custom requirements, costs and risks rise rapidly. The in-house approach signs you on to a better future --- it's a step in the right direction. The commercial services may or may not be easier today, but at some point you'll want to switch out of them. It may still be worth using them today --- but the long-term perspective is at least a point in Polyaxon's favour.