Hacker News Reader

Comment by nouri | Hacker News Reader

We’re considering adopting DBT or a similar tool for the orchestration of our data pipelines on Snowflake. But we also explored Snowflake Dynamic Tables, and they make it easy to build a complex DAG without having to describe it.

I’m curious if data warehouse features like materialized views or dynamic tables will end up making DBT or the like obsolete?

https://docs.snowflake.com/en/user-guide/dynamic-tables-abou...

View on HN

It is likely that these will make it into dbt since there are many dbt users who also use snowflake. The main thing to consider when deciding to some something other than dbt is:

* can you apply software dev best practices, CI/CD etc * is it proprietary or can you use it with other dbs * is there a large community behind it e.g. dbt packages and dbt python libraries * will you also get docs, dq, lineage or will you need additional tools\ing * will you need to orchestrate other aspects of your data flow e.g. EL, then T, then activation, etc

Databricks also has delta live tables and for the reasons above I usually suggest people consider all of these and not just go all-in with one vendor

This is a great point—in terms of dynamic tables/materialized views, the software engineering best practices ie modularization, testing, version control are not as intuitive/straight-forward to apply in comparison to dbt and Serra. We can also add a direct SnowflakeDynamicTableWriter into our framework to work with the best of both worlds!

Reply on news.ycombinator.com