Airflow and tools like it are probably the biggest reason for the shift. Another issue is the need to integrate different technologies requires having the skills of a software engineer.

When the landscape for tech in DE was oracle, mysql and Cognos, DE's didn't need to know about OOP or consensus algorithms. Because the landscape now includes hadoop, redshift, kafka, spark, airflow, notebooks, TiDB and lord knows what else, DE's need to have most of the skills of a software engineer to be successful.

Not sure what the author was referring to but, Airflow has gotten better / gained wider adoption during that time and my team started using DBT which saved a bunch time and was new during that period.