Hacker News Reader

Comment by dangoldin | Hacker News Reader

Author here and there's nuance here but as a rule of thumb data size is a decent enough proxy. Audience here isn't everyone and the goal was to give less experienced data engineers and folk a sense of modern data tools and a possible approach.

But what did you mean by "Read the first paragraph of the `Cost` section"?

View on HN

>Author here and there's nuance here but as a rule of thumb data size is a decent enough proxy.

It isn't though.

What matters is the memory footprint of the algorithm during execution.

If you're doing transformation that take constant time per item regardless of data size, sure, go for a GPU. If you're doing linear work you can't fit more than 24gb on a desktop card and prices go to the moon quickly after that.

Junior devs doing the equivalent of an outer product on data is the number one reason I've seen data pipelines explode in production.

Yes but most data-heavy tasks are parallelizable. SQL itself is naturally parallelizable. There's a reason Apache RAPIDs, Voltron, Kinetica, Sqream, etc exist.

Full transparency I don't have huge amount of experience at working on this massive scale and to your point you need to understand the problem and constraints before you propose a solution.

You have to revisit the assertion that SQL is naturally paralleliseable. As a guide have a look at the semantics around Spark shuffles.

There are more asterisks attached to each assertion you're making than you can shake a stick at.

There is always a 'simple' transformation that the business requires which turns out to need n^2 space that kills the server it's running on because people believe everything you said above.

Or in other words: most of the time you don't need a seat belt in a car either.

Reply on news.ycombinator.com