Hacker News Reader

roejaon 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

Snowflake has their own sql engine and is more of a serverless option. Databricks started off with spark but now also has a sql engine(optional serverless) as well, they are using spark in the article.

The delta format is Databricks lakehouse file format, snowflake uses iceberg I believe.

Both Snowflake and Databricks also provide a ton of other features like ML, Orchestration and governance. Motherduck would be the direct competitor here.

Saying that there are now extensions to query snowflake or databricks data from duckdb for simple ad hoc querying.

Duckdb is fantastic and has saved me so many times strongly recommended.

roejaon Dbt Cloud announces a change from seat-based to consumption-based pricing

This change will cause data team into the bad pattern of trying to cram as much into a model as possible rather then splitting them up into sub models that can be used for multiple use cases.

From the docs https://docs.getdbt.com/docs/cloud/billing#what-counts-as-a-... `Models are counted when built and run` so running a 200 model dbt project 3 times a day would just about hit the 20k limit a month.

Given a year ago dbt cloud was $50 per seat without usage based pricing, building your own ci/cd system with github actions is the way to go with dbt these days.

roejaon DBT Cloud increase Team plan price by 100% and limit features at the same time

Having deployed DBT Core I am glad I spent the effort to set up self hosting with ci/cd for both dbt runs and docs.

A point of frustration I have is the idea that Data Analysts need a specialized cloud based ide to work in because vscode and git is too difficult to learn. I worked at a large company where analysts would complain the database was down due to the cloud ide falling over and stop work until it is fixed. Meanwhile the group of us using a local ide and a jdbc connection kept working.

Any good data analyst should be able to use vscode and git. Any good data engineer should be able to setup the ci/cd side to the point for the analyst doesn't need to think about deployment.

Analyst writes code -> creates pr when ready to test -> pr builds models into temp location and runs tests -> analyst iterates -> pr review -> merge to main -> deploy to prod runner