We developed a lot of Python scripts. To manage them I created some helper tools to integrate them via make. People are welcome to reuse it, I got it released it as open source software. I named it make-booster: https://github.com/david-a-wheeler/make-booster
From its readme:
"This project (contained in this directory and below) provides utility routines intended to greatly simplify data processing (particularly a data pipeline) using GNU make. It includes some mechanisms specifically to help Python, as well as general-purpose mechanisms that can be useful in any system. In particular, it helps reliably reproduce results, and it automatically determines what needs to run and runs only that (producing a significant speedup in most cases)."
"For example, imagine that Python file BBB.py says include CC, and file CC.py reads from file F.txt (and CC.py declares its INPUTS= as described below). Now if you modify file F.txt or CC.py, any rule that runs BBB.py will automatically be re-run in the correct order when you use make, even if you didn't directly edit BBB.py."
This is NOT functionality directly provided by Python, and the overhead with >1000 files was 0.07seconds which we could live with :-).
Make provides a way to handle dependencies as DAGs. Using it at scale requires that you call or write mechanisms to provide that DAG dependency info, but any such tool needs that info. Some compilers already come with generators for make, and in those cases it's especially convenient to use make.