Hacker News Reader

Comment by soumyadeb | Hacker News Reader

People are excited about Snowflake because it can completely disrupt the traditional data-warehouse market.

The legacy players like Teradata and Exadata (from Oracle) really don't scale. Teradata has ~2B in revenue, Exadata is probably in the same range. That's all up for grabs but that's only scratching the surface.

Historically, only transactional data was dumped into the warehouse. Snowflake is selling storage at S3 price (plus you get compression so often ends up cheaper) while they are making money of compute/query. If they can provide all the right query abstractions (SQL, full-text search), in theory all data can be thrown in Snowflake. Yes, tech savy bay area companies can setup their own stack using Presto etc but rest of the world is not like that.

View on HN

[Teradata employee here.]

> The legacy players like Teradata and Exadata (from Oracle) really don't scale.

I get why Teradata gets labelled "legacy", but one of Teradata's main differentiators is scale. Teradata engineers have been tackling incredibly interesting scale problems (on many dimensions of "scale") for 40 years. Teradata has many customers who routinely manage and perform analytics on many petabytes of data.

> Historically, only transactional data was dumped into the warehouse.

That was once true, because initially that was all the data that companies had. However, companies have long since used data warehouses for all kinds of data — sensor data, text, behavioral data, product info/BOMs, vendor info, contract info, etc. — whatever's necessary to run the business.

> Snowflake is selling storage at S3 price…

This is important, but not unique. For example, Teradata's current product has native support for S3 and S3-compatible object stores, and you can query them just like any other database table, join that data with data in high-performance native storage, etc.

Sorry, I didn't clarify well. I am sure it scales technically well but not on cost.

My experience of TD is > 10 yrs and then the multi-node version was substantially more expensive than the single-node version. Also, storage and compute was coupled which meant I had to pay for nodes even if 99% of my data was cold. That's a problem with RedShift too but not for Snowflake.

De-coupling storage and compute was a brilliant move by Snowflake. BigQuery can completely abstracted compute - you don't provision compute and only pay for data scanned. However, it gives you a sense of insecurity around cost - A single bad cron job running a query every sec can blow up your cost (real-life experience). Snowflake provides the best cost/performance tradeoff I have seen.

> I am sure it scales technically well but not on cost.

The honest answer is "it depends". Because Teradata is a different beast, per-query pricing can be significantly cheaper than Snowflake with high-volume workloads. It's worth trying both to evaluate cost and performance.

> Also, storage and compute was coupled which meant I had to pay for nodes even if 99% of my data was cold.

Yes, it used to be that everything had to into Teradata's high-performance filesystem. These days, Vantage's native object storage support means that you can keep that cold data in S3.

>>However, it gives you a sense of insecurity around cost - A single bad cron job running a query every sec can blow up your cost (real-life experience).

Only if you're using on-demand. Instead reserve some slots and pay flat rate. The minimum quantity is very low, and the minimum time is 1min.

> Teradata's current product has native support for S3 and S3-compatible object stores too, and you can query them just like any other database table, join that data with data in high-performance native storage, etc.

Storage costs for S3 (or any cloud-provider object storage) are only one dimension of the price. The other is interaction costs which can get prohibitively expensive, for example if you accidentally forget to provide a partition key in your query predicate. Snowflake absorbs this cost if you use internal storage (or just copy into tables).

Snowflake doesn't absorb the cost because there is no cost.

The benefit of native tables for all columnar databases is that it provides an optimized format with metadata for each column, which is then used to eliminate most of the data retrieval during query time. The more selective your query, the faster the results.

> Yes, tech savy bay area companies can setup their own stack using Presto etc but rest of the world is not like that.

My last company was an early adopter of Snowflake. And we tried Presto first, circa 2016 and Presto was sloooow. We were using vertica at the time and it was so much slower. Snowflake on the other hand was able to perform on the same order of latency as Vertica, which was pretty crazy to us.

That's interesting. I thought Vertica's pitch was real-time analytics for which draditional disk based data-warehouses are too slow.

Vertica is a disk based analyctics database. It was very fast, but also very expensive. And hardware failures could be particularly difficult to recover from.

Vertica was very powerful for us but the separation of compute from storage was a critical feature of SF that motivated us to switch. We evaluated EON mode but SF was too easy. But from performance were also running nightly batch processing on a 22 node Vertica cluster that was taking 6 hours per night to run on highly optimized projections. We threw the same query at SF and 8XL cluster and it finished in about 30 minutes. The biggest difference however is cost where we are spending more on SF than we otherwise would have on Vertica.

SF prices may have raised. We had a 48 node cluster in AWS, which wasn’t cheap obviously and our license was also seven figures a year. And we got a good deal from Snowflake because we were an early large customer. One big advantage for us was that we could auto scale snowflake based on our load to save more money.

So why did you switch from Vertica (or did you)?

Vertica was too expensive, their licensing fees were terrible at our scale. Operations were also awful, if we had two nodes go down we were always in trouble. We built an EBS solution that made it a little better, but it still wasn’t tenable long term.

Good info -- thanks.

Their Eon mode product is very similar to Snowflake, with S3 storage and semi-dynamic compute nodes, but they may not be as slick at marketing it or providing a UI.

Yes Eon is similar BUT it is not nearly as turnkey. Agree it is not well marketed.

My previous company still uses Vertica (on-prem) but it still wasn't as fast as it should be. Trying to hire anyone for an operations team was an enterprise in futility, since it's a rather niche tech. Maybe it would've been better in the cloud, but here large companies are cautious about vendor lock-in and after 2014, potential impact of sanctions.

>>>Yes, tech savy bay area companies can setup their own stack using [insert open source tool here] etc but rest of the world is not like that.

It sounds like a ton of these cloud infra companies have this product strategy (datadog, snowflake, elastic, hashicorp, etc)

When you look at cloud infra companies like that, their competitive advantage is in quickly being able to ingest data and make it accessible, so an off-the-shelf solution likely doesn't exist for their particular use-case. Also, since that operation is your competitive advantage, you should look to in-source it rather than reach for a COTS solution.

Ironic that products named exadata and teradata (guess we skipped petadata) dont scale.

Smaller companies with presto won’t get the same performance benefit.

Snowflake & BigQuery get the ability to have multiple customers on a large cluster.

It’d be cost prohibitive for a single smaller customer to have all that compute sitting idle for a few queries per minute.

Storage also benefits as snowflake/ BQ can shard your data across a much larger array of disk giving you better IO.

Think is it faster to drive a car 100 ft starting at 0mph and flooring it. Or to drive 100ft with a car which starts off doing 120mph

Reply on news.ycombinator.com