Also, I can write my T in plain ol’ SQL (granted, with some jinja) instead of this dbt-QL that I can’t copy and paste into my database console or share with a non-dbt user.
So, folks who have adopted dbt: what am I missing by being a fuddy-duddy?
It sounds like in your approach this would be writing this dependency logic into each DAG you schedule on airflow.
In the same way you would interpolate your jinja SQL before copying it into the database, you would use dbt compile or the output from a dbt run from the target/ folder and copy that SQL into your DB console or to share.
EDIT: This means your T is a single airflow node in each DAG, though I then still use airflow for the E/L tasks around it
- Automatic DAG generation based on dbt-QL declared dependencies.
- The structure of where (db/schema) and how (table/view/temporary) things are built is defined in a YAML configuration, not the individual SQL statements.
- Testing/documentation baked in.
Sure, you can manage every select statement as its own task, but it becomes pretty infeasible once things scale.
dbt can still be administered alongside all other E and L-type tasks. It's just a Python CLI wrapped around SQL SELECT statements.