The problems in this article and in the comments are some of the stuff we have heard at Magniv in the passed few months when talking data practitioners. We are focused on solving some subset of these problems.

Personally, I think Airflow is currently being un-bundled and will continue to be with more task specific tools.

At the very least, if un-bundling doesnt occur, Prefect and Dagster are working hard to solve lots of these issues with Airflow.

Evolution of products and engineering practices is not linear and sometimes doesnt even make sense when looking at a-posteriori (as much as I would like it to follow some logical process). Will be interesting how this space will develop in the next year or so.