Airflow 2.0
airflow.apache.org
airflow.apache.org
Jenkins is also a generic job scheduler, but it is primarily used for CI/CD. It seems like there's some overlap here though; I'm interested to hear other peoples thoughts on "I have a bunch of jobs that need to be run; they aren't necessarily ETL or CI/CD; what tool is good for this, and how would I differentiate between them."
It seems that mostly people use whatever they have setup, so if someone already does CI/CD through jenkins, they will add other unrelated jobs to jenkins just to avoid setting something else up. Any thoughts on if things are being left on the table here, as even generic tools can make you feel like you are "swimming up stream" if you are doing things that the rest of the community is not with them.
Here is my personal opinion of some Pros and Cons for the types of jobs that are in the fuzzy area of overlapping functionality.
Jenkins gui and interface handles lots of standalone jobs more nicely.
Jenkins has much better support for parameterised jobs that can be kicked off manually.
Airflow can handle dependencies between jobs in a much better way. Nicely defining and visualising dags of job really is the killer feature.
Browsing task/job logs is nicer in Airflow IMO.
Airflow scheduler is flaky - hopefully better in 2.0.
Airflow has much more “magic” than Jenkins this is often infuriating.
All that said, my preference is to move jobs to Airflow. Getting a nice gui for manually triggered jobs in airflow is/was the only large missing piece for me.
So there is a GUI, but it isn't nice. Am I reading that correctly?
Or perhaps the functionality doesn't exist at all? Or can be done at the command line or API?
The UI is fine, but looks to be improved in 2.0.
Main improvement I am looking forward to are the scheduler changes, I run a few jobs that trigger every minute in airflow, and it's definitely not intended for that.
Edit:
Sorry I misspoke here
The only thing that is no longer allowed is using a bitshift operator between a DAG and a task.
task_1 >> task_2
my_dag >> task_1
The only thing that is no longer allowed is using a bitshift operator between a DAG and a task.
task_1 >> task_2
my_dag >> task_1
Is there an official guide of how to update from 1.10.x to 2.0, or do we still need to do the 1.14 update, then move over? I'm interested in updating ASAP, but scared to break my production.
Feel free to AMA about Airflow's new features/the roadmap going forward!
Would be perfect to have separate dependencies for different DAGs, otherwise we always end up with a pile of everything ever needed with no clear way to remove obsolete packages from setup.
1. If you're using the KubernetesExecutor, you can point to custom images for individual tasks, this will primarily work if you're storing DAGs in git or a volume (or if you want to handle baking in DAGs for different images).
2. You can use custom images in KEDA queues. This way you can simply point to a queue for all tasks in that DAG and they will run in that environment.
3. You can use the k8spodoperator. Now that the k8spodoperator allows for templating, it would be pretty easy to create a template for a pod and just inject different commands for different steps.
Hope that helps!
That's not exactly what I was looking for, though. Because every listed approach injects technical complexity in the middle of my business logic.
I.e. if I have two consequent tasks I have to define them as a separate scripts or commands, package them, upload and then orchestrate them in a completely different place.
While in Prefect I have all the niceness of writing almost plain Python (as with new tasks API in Airflow), then I can package and distribute the whole thing in docker image with a single command. It really matters!
We've found Airflow and ECS Fargate to be a great combination for running ETLs. It keeps Airflow small and dumb, and lets the Fargate containers do the heavy or complicated lifting in language of developer's choice.
We'd really appreciate if the ECS Operator could be given a bit of attention:
Running a task on FARGATE_SPOT containers is a cheap, convenient option, but it requires passing capacityProviderStrategy in. https://issues.apache.org/jira/browse/AIRFLOW-6604
Also currently the ECSOperator only shows the output logs once the task has finished (which could take hours), it'd be better if the operator could poll the Cloudwatch logs during the run rather than wait for it to finish.
---
Congratulations on the release, I'm looking forward to upgrading soon, and trying out the new features and syntax!
One really nice feature of 2.0 is now the "providers (hooks, operators, etc.) are released separately from Airflow itself. So you won't need to upgrade airflow to get improved AWS operators unless there is a breaking change.
[1] https://aws.amazon.com/managed-workflows-for-apache-airflow/
I was curious about it but the pricing page scared me off, the smallest which runs 50 DAGs is about $0.49/hr! I couldn't understand why the pricing was that way.
THIS! Mainly looking forward to the scheduler HA improvements! That has been the biggest pain in a self managed Airflow environment.
If I may hijack this thread for a small feature request plea. I wish that the gantt chart is exposed as an API.
Or, simply support alerting directly inside the gantt chart.
It would make monitoring and alerting much much simpler.
could you make an airflow issue related to that or start a thread in the dev list? That could be interesting! (though you might want to wait until after the holidays as we're all a bit wiped :) )
I was more interested in event-based tasks though. For example, a sale happens, and then a report about that individual sale gets generated. At the time, this seemed to require some Airflow hacks and would cause me to miss out on some of its features. Is this still the case?
https://user-images.githubusercontent.com/3267/96045809-852d...
https://aws.amazon.com/managed-workflows-for-apache-airflow/
I seriously love that you're an Airflow user since your spark talks first got me into OSS.
And Airflow is designed to spread tasks between a set of available workers, so while you can make jobs that trigger something remotely through SSH (for example) I'm still looking for a way to have a 'remote worker' that runs airflow jobs on the remote system itself rather than through an SSH connection (mostly because of a rather peculiar use-case where we'd prefer to run jobs on a remote network without two-way communication).
If you're comfortable with docker-compose you can probably find an example setup and get it running in a few minutes though.
ASF communities work hard to be open to contributions from any and all parties, which is important for avoiding alienation of minority factions and keeping the community from fragmenting, and also for maintaining project independence and not getting captured by a single dominant entity. As a side effect, contributions are often negotiated but rarely rejected, which exerts pressure in the direction of feature bloat: the acute needs of the contributor tend to outweigh the diffuse benefit of simplicity.
Such pressure is not impossible to resist, but it usually takes dedication and and superior negotiation skills from core maintainers. A modular architecture is also very helpful for accommodating contributors without compromising usability, which is why so many successful ASF projects use plugins.
Airflow is a backend project built by backend engineers.
Most UI people don't use Airflow or know what it is.
@ryanhamilton is the first front-end dev to become a committer on the project and that JUST happened a few months ago.
Of course a lot of people think of Airflow as 'simpler' than Luigi, because Airflow is all graphical. YMMV.