Hacker News Reader

> We're building a visual pipeline tool for data scientists.

As a Sr. DS/ML Engineer, this doesn't speak to me.

Yeah me too. I think visualizations are cool and can be useful in terms of monitoring the progress of analysis runs. But I can't see my self using a GUI for setting up pipelines when I can do it more explicitly through scripting or some kind of a command line based pipeline framework.

However, maybe the use case here is for team collaboration where the pipelines are easier conveyed through visualizations.

There's a large tradition of data analysis/BI tools using visual programming/ "no-code" tools, think Talend, etc. They're popular since data analysis typically contains a number of people without coding backgrounds. Even among data science teams, many don't have the engineering background of data engineers (and certainly there's a segment of data scientists who don't care as much about building the data pipelines and the cleaning that's necessary).

Low code tools _can_ (but not necessarily will) effectively empower data teams.

Thanks for your input. We are planning to also support an API and/or CLI (besides the GUI) to set up the pipelines through scripting. Depending on what we hear from the community we will definitely add this sooner rather than later.

You make a great point for the use of monitoring the progress of a run. During pipeline runs the visual pipeline interface (also in view only mode during or after running pipeline runs as part of an experiment) can be helpful as a visual indication of where the pipeline failed, how long pipeline steps took to execute and whether certain pipeline steps are still running.

Lastly, for team collaboration we think the visual interface is indeed helpful to explain (especially to the less technical people) the structure of the code, what exactly the pipeline sets out to solve and how the pipeline solves it. In addition, the possibility to add intermediary pipeline steps that serve as dashboards can be helpful for collaboration.

I personally agree that this does not seem too useful for any DS team that needs to deploy a model to production. But there are a whole horde of DS teams whose outputs are basically PPT presentations.

Think like pricing forecasts, decision modelling, marketing segmentation for product design, ....

I think a common thread to these teams -- at least those that I've seen -- is that they consist of stats->DS backgrounds, and no eng->DS backgrounds.

Many of these teams are orchestrating everything within the notebook. I've seen notebooks that contain complex workflows that extend to 10K LOC. I've lost sleep over such things.

I agree, and so many data science and data engineering tools all seem to revolve around using notebooks, much to my frustration. I’ve worked in places whose data pipelines were built around seemingly infinite notebooks, all containing consistently poor software engineering.

It’s been enough to make me vow to not let people write notebooks that go into prod under my watch lol.

I’m constantly on the watch for software engineering focused tools that solve the issues, rather than data science/engineering focused tools. So many are inextricably linked into python as well, which doesn’t gel nicely with anywhere that has multiple languages in the codebase.

We'll allow your team to move parts of your code base from notebooks to scripts (.py/.sh/.R) to alleviate your frustration.

That way you can keep using notebooks only for those parts where it makes most sense.

They come together in pipelines that are JSON defined and git-versioned.

Shameless plug, but I help maintain Cortex, and "software engineering focused tools that solve (ML) issues" is a neat summary of our entire philosophy. For example, instead of notebooks, our model serving platform (https://github.com/cortexlabs/cortex) uses YAML to structure deployments, and Python scripts to write inference APIs.

It's still inextricably linked to Python, but only for writing your API. It's agnostic as to how the model itself is developed, so long as it can generate predictions.

Currently, we are not focused on helping DS teams putting models into production.

It's those 'messy' projects with 10K LoC notebook that could easily be broken up into multiple steps (perhaps some of them notebooks, some of them Python scripts with library function usage) that we feel are a great match for pipelines in Orchest today.

When a team is still experimenting with what models to go with (i.e. trying neural networks v.s. decision trees) it can be helpful to have a more structured prototyping environment with reproducability and easier scalability. Which is also where Orchest shines.

If you want to test some of these models in production, you could easily push artificats to endpoints for serving, in the final steps of a pipeline.

Maybe we need a better way to explain what we do. After reading through the comments and getting a better picture of what we're providing, does Orchest still not speak to you?

Would love to learn more, as we're particularly keen on understanding how to communicate in the clearest way.

Disclaimer: I've recently at my company built and released a product that has a displayed DAG on screen and allows people to build ML pipelines in addition to the existing data prep and blend capability. My company is profitable and the ML product is doing well. We also have the ability to have jupyter notebooks execute in-line with the DAG.

The comments on here are reflective of the fact that selling products like yours to typical coders is a dead-end. They don't like it or want it, not because it wouldn't be useful to them, but because they are attached to their current tooling and don't see a need for making their workflows more maintainable for non-coders. Some coders get it, after working in an office and realizing they are spending inordinate amounts of time tweaking their software for various data sources instead of focusing on the actual interesting stuff, but most don't.

Your product has the challenge of falling in a valley between the domain experts who can't code and therefore won't have notebooks available to pull into each node, and the people who can code and have notebooks not wanting to spend money or use a dirty, filthy, evil GUI instead of their precious glue code.

Your sweet spot of customers won't be on HN. They are the massive army of people with the job title "data scientist" who really aren't close to being actual data scientists. The enterprises are filled to the brim with them. Many of them can barely write Python or R, and are outright frauds. But plenty of them have a cursory knowledge of Python coding in Jupyter, and can actually do some Kaggle problems. However, basic ETL skills, web scraping, etc are foreign to them, and they have no ability to embed their shitty Jupyter notebook code into an ETL pipeline. These people aren't on HN. If they were, they wouldn't be so shitty at their job. Those people are your customers. The underqualified, borderline fraud data scientists who are EVERYWHERE. I sound like a cynical asshole.

But there's a silver lining here:

The best customers you can ever get aren't in that camp at all, and also would add a lot of value to their orgs using your tool. These folks are the domain expert analysts who really, really know their company's data and have the deep domain expertise. They are the people who data scientists have to talk to to be able to create value. You want to go after these people, and upskill them to basic python. People who know their business and it's data, and have just a little bit of data science skills create far more value than the opposite.

Thanks for sharing your thoughts. I think you accurately identify some of the key challenges.

We really like the idea of empowering the domain experts to leverage data by upskilling them to Python. It's a much better tool than i.e. Excel, which tends to dominate many corporate environments still today.

However, we'll also make sure Orchest plays well with the tools software engineers already use (such as git for versioning, framework/library integration, and Bring Your Own Editor support).

Reminds me a bit of https://plynx.com/ , and it's also open source. Is there a major differentiator I'm missing? Also, what is your idea regarding the use case. Why would I need to run it locally for example? Is it mostly about productionizing ML?

Similarly, there seems to be partial overlap with MLFlow for tracking iterations.

I would find a comparison table vs. existing tools useful, to help me consider Orchest by placing it in my existing workflow.

We want to try to make it easier for people to understand the landscape of tools and our position within it.

I personally like something like GitLab's https://about.gitlab.com/devops-tools/. We'll try to put up something similar on our website at some point.

I'm not deeply familiar with MLFlow, but from what I have seen/read it is more of a tracking framework that you can integrate into an existing codebase.

While Orchest allows you to take your existing codebase and structure it into a pipeline to get a visual and containerized way of interacting with the codebase (allow a mix of notebooks and .py/.sh/.R scripts), running the pipeline, and visually inspecting success/failure of pipeline runs/steps.

Another key point of difference is how we are more concerned with managing the flow of data. Since we let you build pipelines we can give you abstractions to separate data flow from the pipeline code. I.e. letting you define generic pipelines that take any data source (in some standard form, like a schema'd database) and produce reports. Because we control the data source in relation to the containerized pipelines we can also make sure the whole thing performs well when it's being executed in parallel (i.e. same version of the pipeline running grid search over paramaterized pipelines). In other words, we also control more of the underlying infrastructure when executing the pipelines.

I hadn't seen Plynx before, and indeed it does look quite similar.

Something that I believe we do that I don't see with Plynx is our support for "debugging" pipeline in realtime through extensive Jupyter integration.

When a pipeline step is a Jupyter notebook, you can execute previous steps in the pipeline (partially execute the pipeline graph). It passes data to the running notebook kernel and which allows you to explore it interactively. This makes it a lot easier to guarantee your incoming data matches your expectations.

We also put each step in a pipeline in a separate container (with its own image), which will simplify a lot of things when taking the effort to productionize pipelines to make them suitable for horizontal scale-out.

Edit: running locally can be useful if you want to use your own resources (i.e. your own GPU-rack) to run compute heavy data science pipelines. But we envision for most real-world team usage they'll want to run in the cloud. We're working on making it run directly on top of cloud provided Kubernetes engines.

This looks cool! A couple of questions:

1. Currently, if I install something in the notebook, does it get re-installed every time the pipeline is run? Is there any way to "snapshot" the state of the container?

2. Where is the data stored between the steps?

3. How well-integrated is it with AWS cloud primitives such as EC2 instances, EFS, and S3?

Thanks!

1. Right now additional dependencies for the container need to be re-installed whenever you run the pipeline. During the entire Jupyter kernel session though, the container state and thus any installed dependencies remain available. We're working on either supporting container snapshots or custom container images (with desired dependencies pre-installed). We'll likely go with snaphots as they'll be easier from an end-user perspective.

2. During step execution data is stored inside of either the pipeline directory (which contains for example the .ipynb/.py/.R/.sh files) or in any of the mounted directories (through data sources).

When you run the pipeline as part of an experiment a copy is created so that any state generated by any of the steps inside of the pipeline directory is isolated from the 'working copy' of the pipeline.

Edit: forgot to mention that we support memory-based data transfer between steps which is faster and doesn't "pollute" your pipeline directory. It does require your data to fit in memory though. We use Apache Arrow's Plasma for this.

3. AWS S3 and AWS Redshift are currently supported as data sources. Some light docs at https://orchest-sdk.readthedocs.io/en/latest/python.html#dat... (to be improved!) and the relevant SDK source (https://github.com/orchest/orchest-sdk/blob/master/python/or...). We should look into EFS. Do you have a use case in mind?

Congratulations! I remember your earlier project: grid studio. Do you support scheduling periodic tasks? Do you support execution triggered with webhook? or some way to expose notebook as REST API?

Grid Studio is indeed the project Rick worked on before starting to work on Orchest. Great to hear that you are familiar with it.

Currently, tasks can only be scheduled to run at a set time. Could you elaborate a little on why you would want tasks to run periodically? We have some ideas on why this might be helpful, but would love to hear your take on this. (Periodic task scheduling is absolutely something we can add.)

The front-end of the application actually makes calls to our API for multiple things, among which execution of tasks (our internal endpoints can be found here https://github.com/orchest/orchest/tree/master/orchest/orche...). Exposing a (REST) API to the user to interact with the pipeline and start executions is on our roadmap.

I hope this answers your questions.

Scheduling periodic tasks can be useful for creating ETLs or to create alarms (every 1h hour check data for new values if condition is meet then send the email alert)

Agreed! ;)

Thanks for sharing, this is super helpful. I'm endlessly jealous of the teams at Uber and Booking and their fancy tools

We'll make sure the rest of the world gets those great tools too!

Really cool! I can’t wait to start playing with it.

Can two people collaborate on the same project at the same time?

That's great to hear! Right now it's not fully supported to edit a pipeline at the same time. We're moving towards a git-based async collaboration approach where you can fork and merge pipelines to make sure changes you make to code/Notebooks aren't going to surprise you in your analysis/models.

How do you think about this compared to something like Dataiku?

Great question. We think a key point of difference is that we'll never focus on providing a 'clicky' way of building actual data processing, training, transformation steps.

For example, in Dataiku you can define and transform columns by using a GUI. We never saw that as more productive than writing transformations using i.e. R data frames, Pandas or Koalas (Pandas on Spark). The Python/R scripts that do the actual transformation can be cleanly versioned, re-used, and modified in a much nicer way than anything produced with GUI based transformation/processing.

You also don't ask people to invest their time and skills into a way of doing things that is specific to a particular tool. I.e. if you can write great data transforms in Pandas today you don't have to change anything when you start using Orchest to build your data pipelines.

I have wanted something like this.

Julia support?

Yes please! It's on our roadmap. We want to be language agnostic as much as possible close to the spirit of Jupyter.

We have a lot in place already to make it easy to add languages, so if more people like it we'll have Julia support coming up relatively soon.

Show HN: Orchest – Data Science Pipelines