But "naïve programming" (the art of writing a dumb algorithm with no regards to performance or optimizations) can bite you hard: https://docs.dask.org/en/latest/best-practices.html#avoid-ve...
I implemented an algorithm that was slower than the non-parallelized version this way :D
Then they gave us a sample of a dataframe, to analyze firstly with Pandas, and then with Dask, and compare. The times not only didn't improve, or even equal Pandas: they got worse by more than 10x. The teachers looked the code with a mix of disbelief and horror, and only after that they started to talk about tradeoffs, overheads, data fitting in memory, etc.
This means that nothing happens until it needs to. So I have experienced .head() taking hours if there's a lot of upstream processing.
Another reason for this could be that your file is distributed across many nodes, and thus you need lots of network requests to get the file.
I'm not disagreeing with you that Spark can often have problems, but it's a really useful tool that can do things that are really hard to do elsewhere (like distributed ML).
It's just a really unfriendly system for creating packages for everyone to run on a single node unfortunately.
I definitely get the frustration, but waiting for sklearn for hours to fit a fraction of the observations that spark can do is also frustrating, in a different way.
> run on a single node
yeah, Spark is pretty crap here, but it's not really one of their main use cases, to be fair.
amen. people should learn to write fast code instead of "spraying and praying"