my thoughts, exactly, for around 10 years, since my first serious data engineering projects. So many projects I worked on, all suffering this problem: complex individual data pipelines.

Sometimes I wrote small frameworks on my own, sometimes I used 3rd party tools. I also tried to create a universal framework for that, trying to solve exactly what you described. Never finished it, because I am more of the no-finishing guy.

However, at some point I came to the conclusion, that it's simply not possible. which is nonsense and more of an excuse. as you said, it's just data in and data out, processed through some simple rules.

so, I'd know the concept, but there are a million other projects in my head.