580 karma · joined December 5, 2015
> The top exports of Australia are Iron Ore ($79.6B), Coal Briquettes ($36.4B), Petroleum Gas ($26.8B), Gold ($17.7B), and Frozen Bovine Meat ($4B)
[1] https://en.wikipedia.org/wiki/Phone_connector_(audio)#Miniat...
Their recommendations used to be world class until they got rid of the user feedback mechanisms.
I agree with you, a rewrite is probably how they should have tackled this one.
It sounds like in your approach this would be writing this dependency logic into each DAG you schedule on airflow.
In the same way you would interpolate your jinja SQL before copying it into the database, you would use dbt compile or the output from a dbt run from the target/ folder and copy that SQL into your DB console or to share.
EDIT: This means your T is a single airflow node in each DAG, though I then still use airflow for the E/L tasks around it
We use DBT to manage the DAG for the BQ transformations, put this in a container and deploy it into the kubernetes cluster that airflow is running on as a single node.
Airflow can then handle the scheduling and DAG nodes for non DWH dependencies such as loading/checking for files, kicking off tasks that need to run after the DWH refresh and the like.
I find once it is set up it is extremely easy for small teams to follow the pattern, and the single view of all the pipelines running is a great benefit - as well as handling the logic around last successful runs etc., that would need to be implemented manually if using simple cron jobs.
I guess disappeared could mean they are on a beach somewhere under an alias, but I don't think we can assume that for all of them