Kedro – Creating reproducible, maintainable and modular data science code

No pros or cons on deploying models. Seems like that is a significant part of the functional value?

My experience with McK came in two phases. Phase one started with Kedro, and phase 2 was a cash grab, move as hot and fast as you can. In my experience phase 2 got off the ground quickly, but after week the notebook was riddled with run these sections, but not these, 4 devs on the team were hanging out having coffee much of the day because they were waiting for time in the notebook to implement their changes. There was no version management so days were lost to, well someone deleted something they shouldn't have now we need to rewrite it. The lack of code review, linting, and formatting tools left them swimming in messy code that they would clean up later, but later never came.

The kedro project is still running nearly 3 years after it started, the notebooks are long forgotten.

Notebooks are fine for single contributors if that is what they are comfortable with. If that is what they are comfortable with its probably because they have not experienced engagements like you are bringing to them that require them to collaborate as a larger team. If you plan to effectively run projects that last longer than a few weeks with more than one data scientist, I'd really challenge them to lean into kedro. The long term productivity of the project will greatly benefit from it. You will be showing the team a more sustainable way of creating pipelines that will lead to continued success after you leave. This leaves a better name for you than a quick cash grab that gets long forgotten in my opinion.

I completely agree with the cons you outlined, especially your point about "productivity drops when data scientists leave notebooks."

A few years ago, I started working as a data scientist at a big financial firm and reviewed all workflow orchestrator available tools (including Kedro). I didn't like that all of them forced me to re-write my Jupyter code into their frameworks (they're supposed to make me more productive, not less).

True, notebooks have their issues but they can be fixed (I don't buy that "Jupyter is only for prototyping argument"). So, long story short, I started a project with a friend that makes us more productive by fixing the problems that notebooks' problems. https://github.com/ploomber/ploomber

Really interesting response. While there's absolutely much to love about Kedro (we spoke to the core dev team - they're a fantastic bunch), we created Orchest to counter the cons you mentioned.

https://github.com/orchest/orchest

What do you think of Orchest? Given your unique perspective we'd love to hear what you think of it.

We've been having success with agencies, especially when the hand-off needs to be something the clients can easily run with.

Am from McK not involved with Kedro, it’s been open source for awhile now and I don’t think we are abandoning it, but it was apparently donated to the Linux foundation.

As far as impressions of it as a tool go it’s really just an opinionated way to build a pipeline and structure a project. Which is pretty useful when you have many. If you’re doing something that you know will be one off it is a bit overkill. Like using airflow when a makefile will do. There are some pretty nice plug-ins though if you want to try and have an opinionated ecosystem.

Hello maintainer here - hopefully some of our open source users chime in, but my favourite will always be NASA: https://github.com/nasa/ML-airport-configuration

Donation is the normal term for proejcts joining the foundation - in doing so you establish a steering committee where members are entitled to voting rights. To graduate as an incubation project, 5 organisations need to join your board and the project is thus the priorities of the project will no longer be driven by just one organisation.

In the short term there is still a full time internal team staffed and maintaining the project, but excitingly we now have a mechanism for new collaborators to properly come onboard.

Worked with McK on a modeling tool built with a data pipeline in Kedro - it's very opinionated as to how the pipeline has to be built, and I was not super convinced that it would be useful to use it in any other context (as in, outside of a collaboration with McK). Also, YMMV, but I have never heard of any company doing ML or data pipelines using Kedro on their own.

I use it quite heavily at my day job. The core pieces of kedro are indispensable on large projects (Pipeline, Catalog, DataSets, Runner, hooks). The marketing has a lot of focus on the template, It's a great template, but not the key feature that brings me back to it. Really like that I never have to worry about io, the catalog abstracts this away. I don't worry about long runs to get started with my work. It saves all intermediate steps automatically, and the Pipeline (DAG) lets me run just the parts that I need for my task. Before using kedro I would need to scaffold out a way to save all of these intermediate steps, or suffer from long run times to get started on my task. Lastly the DAG is indispensible during environment migrations, it can quickly tell me all of its edges that I need to be concerned with picking up and moving with me.

Unlike similar projects kedro is just a python framework. It lets me build, deploy, and orchestrate however works best for my team.

It’s a framework to organize data science code into pipelines.

E.g. if your default might be a series of Python files connected by a master script and some config files, Kedro organizes them into an explicit Pipeline object. Think loading, cleaning, feature gen, model training, mode predictions, etc.

It might be opinionated but it's no less accurate.

Data science is many researchers first foray into programming in any language and, speaking from my own path, was a relatively self-directed discipline in the early days.

I often longed for a 'standard approach' but many of the projects that suggested them were packed full of boilerplate that made code even harder to read and write, especially if your background wasn't heavily into OOP.

Maybe a better word is that it comes with an opinionated template? The core parts of Kedro can all be used individually as you wish. Only want a DAG, you can use just the pipeline, Want a catalog setup so you can store intermediate steps, use the Catalog. You can even make kedro project using all the features in a single file with surprisingly little boilerplate.

You should be able to do that natively in GitHub with lists!

https://docs.github.com/en/get-started/exploring-projects-on...

And Prefect

Kedro sort of fits into a niche where it just overlaps somewhat with 'orchestrators' like Prefect, Metaflow, Dagster, Airflow and others. What makes it slightly different is that it it is focused on the rapid development journey to production, providing guardrails for teams to co-develop ML projects in a way that nudges software engineering best practice and clean code.

The 'finished article' in many cases should be deployed in production in one of those tools which provide specialised bells and whistles like scheduling, monitoring and observability.

Regarding MLFlow, there is also a slight overlap in terms of experimentation, but not things like model serving. Kedro has a mechanism to track experiments, but it's more designed to give users with zero infrastructure something for free out of the box. It's been built in a way that it can be repurposed for more dedicated experiment tracking tools - the folks at neptune.ai built their own plug-in for this purpose: https://docs.neptune.ai/integrations-and-supported-tools/aut...

I believe essentially all of the tools mentioned here are focusing on the engineering persona and not the Data Scientist. Writing classes and functions isn't the jargon of a Data Scientist. At Ploomber we tried to put the Data Scientists in the center of everything, helping them to work together with OPS. Check it out! https://github.com/ploomber/ploomber

Some users have reported the Targets library is similar for that ecosystem

We've built the Ploomber to work with R as well comparing to others who believe only python is for ML. Ploomber is an open source tool that can integrate with R and R studio out of the box. Check it out! https://github.com/ploomber/ploomber

I use snakemake to call up r scripts in pipelines