I built an ML pipeline in pachyderm years ago, maybe 2016, and loved using it then. But my data was small-to-medium, and I stuck with CSVs because back then you couldn't get many of the benefits with a binary format.

I think it's a shame it took Airflow and similar several more years to realize "each step is a docker container" is the right way to build a dag. It's not clear to me why Pachyderm was left behind while Prefect and Dagster became serious contenders, and Airflow/Astronomer started recommending everyone use it just like Pachyderm (container per step).