Hacker News Reader

Comment by moneywoes | Hacker News Reader

Further, what is it different from snowflake that they do?

The biggest difference to consider is storage, I guess. Databricks clusters can store in S3 or wherever you want really. Snowflake has their own storage layer and it always lives in their cloud. This precludes any on-prem deployments.

You can’t be more wrong? Snowflake also stores in s3 or elsewhere, in fact it’s like the only common feature. Snowflake uses proprietary formats mostly though and only gives a sql interface. Think of it as Apple vs android.

> You can’t be more wrong? Snowflake also stores in s3 or elsewhere

Snowflake stores data in S3 in Snowflake's various AWS accounts. Egress fees are necessary when performing anything outside of Snowflake.

Databricks operates on data stored in your S3 on your AWS account(s). Databricks also runs on your compute contained within your AWS account(s).

Both approaches have valid use cases.

> Snowflake stores data in S3 in Snowflake's various AWS accounts. Egress fees are necessary when performing anything outside of Snowflake.

Egress fees are only if you are unloading into a different region/cloud. If your data is in SF AWS us-east-1, and you unload to your S3 in us-east-1, there is no egress fee.

> Snowflake charges a per-byte fee for data egress when users transfer data from a Snowflake account into a different region on the same cloud platform or into a completely different cloud platform. Data transfers within the same region are free.

https://docs.snowflake.com/en/user-guide/cost-understanding-...

Not in your account.

The compute itself can be in your account. The control plane is usually managed by databricks

Databricks are not on-premises either.

> Databricks are not on-premises either.

Usually, no. But in some cases, yes.

I must have missed the product announcement. Do you have a link, because everywhere I look I can't find it.

You can get most products on prem at a certain price & size. A lot of companies will apply resistance though unless the contract size is right b/c on-prem contracts tend to be less unit-profitable, risky, + unique annoying terms or constraints.

If you * need * it, you find a human to talk to (sales or connect from your network).

Snowflake only really offers SQL (with UDFs) to do data transforms. Spark offers SQL, but also code-based solutions (Python, Scala, Java) that can be used to scale things like ML pipelines and non-SQL transforms.

Reply on news.ycombinator.com