Hacker News Reader

Comment by MrPowers | Hacker News Reader

My feeling is that Spark is better for huge ETL jobs and Dask is better for certain types of model building (because of easy access to Python libraries), but don't have any benchmarks to back this up. Would love some benchmark results comparing processing tens of terabytes on equal sized i3.xlarge EC2 clusters to get a better idea of benchmarks.

Have you seen any good Spark vs. Dask benchmarks?

View on HN

The area that we've seen Dask shine the most is ML. This is because the Python ecosystem has more advantages when it comes to ML and there is better GPU support.

https://towardsdatascience.com/supercharging-hyperparameter-...

https://towardsdatascience.com/random-forest-on-gpus-2000x-f...

Disclaimer: We produced those benchmark. I'm a founder of Saturn (https://www.saturncloud.io/) and we focus on providing Databricks-like capabilities with Jupyter + Dask + Prefect, so I definitely have strong feelings in this area.

I would be interested in seeing some spark v dask benchmarks too. Haven’t seen any yet though.

In my experience, Dask really shines when you implement custom numpy computations that could only be done in spark UDFs. We saw a decent performance difference there, but for common built-in computations I’d imagine that spark has better performance.

Edit: after some googling I found this paper with benchmarks.

https://arxiv.org/pdf/1907.13030.pdf

> Results show that despite slight differences between Spark and Dask, both engines perform comparably. However, Dask pipelines risk being limited by Python’s GIL depending on task type and cluster configuration. In all cases, the major limiting factor was data transfer.

It's also worth noting that with Spark, you can perform arbitrary computation using the Dataset API and operating on case classes.

Reply on news.ycombinator.com