python is slow and single threaded, which is a tangible problem for an ETL, and airflow specifically.
...airflow is good enough for other reasons, but that doesn’t mean ‘not python’ gets off the wish list.
...airflow is good enough for other reasons, but that doesn’t mean ‘not python’ gets off the wish list.
Wouldn't the value of optimization at the framework level, be generally... very little?
There is also a problem with python that it does not do thread-based concurrency, but only process-based ditto, where the processes need to communicate in a less robust way with each other, e.g. via HTTP.
Because of this, in our work (scheduling machine learning workflows on HPC clusters), we started see lost HTTP-connections between workers with Luigi when going over 64 workers, even though these workers were doing nothing else than keeping track of a job running on another compute node.
This lead us to use Go instead, with SciPipe, and it has none of these problems. Go being compiled also means we're catching a lot more stupid typos and such at compile time rather than at run time (like 7 days into a 7 day HPC job), which also makes our life easier.
Python is definitely quite a lot easier to write and read, but the robustness of compiled Go code is hard to beat, and I don't wish to go back to a python-based tool.
this is half true, and importantly, in a workflow tool, the part where python is perfectly capable of being multithreaded while waiting on IO (which releases the GIL) should be quite sufficient.
there are problems with python like its memory usage and with airflow specifically like its at times flaky scheduler (maybe that's fixed in newer versions), but multithreading shouldn't be one in this particular context.
You can (and should) program your batch jobs in other languages than python if you have large amount of data of course.
"Python is slow."
What do you mean by "slow" and what have you done to make Python code run faster? I find for most things Python is fast enough or performance can be improved to meet most compute and data demands.
In the future we will all be using Javascript. Language performance debates are moot.