Hacker News Reader

Ask HN: I've recently started using Spark for some simple stream processing amenable to embarrassing parallelism (that also involves making a REST call per row). It's been a horrible, horrible experience, troubleshooting IT issues and debugging code are extremely time consuming.

Am I missing something or abusing the system or is this the norm?

View on HN

This is normal for spark. We have very little expectation even our senior engineers can optimize their own jobs. It’s a difficult system to learn but can be very powerful.

That said, making REST calls inside spark sounds like a recipe to accidentally DDoS your target endpoint.

Sounds like my experience when tinkering as well.

https://adamdrake.com/command-line-tools-can-be-235x-faster-...

Reply on news.ycombinator.com