AWS Redshift vs. Parquets and Apache Spark

1 point·dancrystalbeach··6 comments

I've been mulling this over, and I can't make up my mind. What's better to use for a large data warehouse, AWS Redshift vs Apache Spark on top of Parquet files stored in s3? There really doesn't seem to be much different to me besides you can probably do "more" transforms with Spark. But the complexity is probably reduced with Redshift. I need thoughts and input.

View on HN

Cost will definitely be a factor but if it's not then I'd suggest Redshift because of the simplicity. Another aspect is how the data will be consumed - Analysts are typically more comfortable using SQL + Redshift. I'd avoid Spark except for very specific use cases as the operational toil is higher with Spark.

the consumption is very SQLish based. "operational toil is higher with Spark" <- I agree with this, this is why I'm leaning towards Redshift. Throwing away half the codebase in favor of some SQL statements is tempting.

Snowflake :) Redshift is also good

See this for comparison benchmarking: https://fivetran.com/blog/warehouse-benchmark

If you want cost efficient high performance Clickhouse is hard to beat

how about the difference in cost? Redshift is pretty expensive. We use Spark to convert our data to Parquet and Presto to query it (although you can just use Spark to query it also)

I've thought about that, i should probably do more detailed analysis of what the actual cost difference would be. I've looked at Presto as well... but again that is just another tech stack to manage.

Reply on news.ycombinator.com