Koheesio: Nike's Python-based framework to build advanced data-pipelines

The idea that you'll just build a tool that makes hiring 10x as many inexperienced devs work is dubious. Just one more new DSL bro. Certainly we have cracked the code that no one else has.

The problem with these types of orgs/tools is that by its nature your DSL constrains the juniors/inexperienced devs to what is currently possible. There is not a lot of learning unless you rotate them through the tools team periodically, which no one does. It's also awful for the devs who are building in experience in something they can't use anywhere else.

I was in one shop where the "tools team" guys epiphany was he would meta-recruit by poaching the best data engineers out of the ETL team, lol. Very explicit "good team / bad team" vibes.

I walked into 20 years of adhoc code that had zero data lineage, recoverability, or scalability that was breaking daily. There were contractors with over a decade of tenure whose job it was to troubleshoot and fix their own brittle processes (and make new ones) daily.

I got laid off (747Max plus pandemic) as I was rolling it out and they went back to the old way.

Subsequently, a new startup (Pantomath) emerged with former GE engineers (and other former colleagues of mine) from my former department to address that problem domain.

Based on my experience trying to socialize this type of solution, sales are going to be a bitch.

From the readme they describe it as "robust" and having a high level of type safety, so I'm guessing they're just leaning towards the "learn about bugs up before they hit production" end of the spectrum than you.

Then again I don't do any of this data engineering stuff so maybe it doesn't matter too much if it doesn't work reliably?

I think this pretty much sums it up, "a structured way".

It's looks to be a thin wrapper around spark to provide a consistent way to structure ETL jobs. They've implemented a mini dsl defining jobs as a datastructure on top of Spark.

I've seen several companies build stuff similar to this internally, defining jobs as a data structure. It all amounts to each company having their own internal conventions, their own view of what is easier for their devs and creating a framework for it. Nike have just decided to make theirs public.

You can do all this simply with simple spark scripts. Personally I'd use simple spark scripts. Big companies with lots of people love making these tools as companies love conventions, their conventions, their style guides, deal with staff churn/on-boarding frequently so believe these kind of things make that easier.

Probably makes sense in Nike as a way of organizing their ETL jobs but that looks to be all it brings. A way to structure/define your simple spark jobs the way Nike devs believe it should be done.

seems to be Spark-only AFAICT?

I’m not against proprietary software, but your website still advertises this product as an open source ELT.

Straw man argument.

>I would expect someone who's seriously writing a new orchestrator in 2024 to cite the alternatives, their shortcomings, and how you intend to address them. Bonus points if you make a neat little table.

Did you even try to read the docs before you launched this critical diatribe?

From the docs (https://engineering.nike.com/koheesio/latest/tutorials/onboa...):

    Advantages of Koheesio

    Using Koheesio instead of raw Spark has several advantages:

    Modularity: Each step in the pipeline (reading, transformation, writing) is encapsulated in its own class, making the code easier to understand and maintain.
    Reusability: Steps can be reused across different tasks, reducing code duplication.
    Testability: Each step can be tested independently, making it easier to write unit tests.
    Flexibility: The behavior of a task can be customized using a Context class.
    Consistency: Koheesio enforces a consistent structure for data processing tasks, making it easier for new developers to understand the codebase.
    Error Handling: Koheesio provides a consistent way to handle errors and exceptions in data processing tasks.
    Logging: Koheesio provides a consistent way to log information and errors in data processing tasks.

    In contrast, using the plain PySpark API for transformations can lead to more verbose and less structured code, which can be harder to understand, maintain, and test. It also doesn't provide the same level of error handling, logging, and flexibility as the Koheesio Transform class.

Or could it be an evidence that the existing tools all have their flaws and any reasonably sized organization will hit these flaws pretty early, so many orgs come to the conclusion that the best approach is to just roll their own that fits their case relatively well?

As far as I can tell from the docs, this is not an orchestrator at all. It is a Python-wrapper for certain runtimes like PySpark. I don't see anything in the docs that mentions DAGs, dependency definitions, scheduling, or deployment.