The Past, Present and Future of Stream Processing

This “article” uses a lot of words to say so little. There is very little to be gained by reading this. It reads like a search engine optimization attempt.

Fully agree. "The Future of Stream Processing is Open Source and Cloud" Isn't that an oxymoron?

Ask HN: I've recently started using Spark for some simple stream processing amenable to embarrassing parallelism (that also involves making a REST call per row). It's been a horrible, horrible experience, troubleshooting IT issues and debugging code are extremely time consuming.

Am I missing something or abusing the system or is this the norm?

Sounds like my experience when tinkering as well.

https://adamdrake.com/command-line-tools-can-be-235x-faster-...

This is normal for spark. We have very little expectation even our senior engineers can optimize their own jobs. It’s a difficult system to learn but can be very powerful.

That said, making REST calls inside spark sounds like a recipe to accidentally DDoS your target endpoint.

Here's a bunch of interesting deep dives: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs...

I've never seen so much bad technology quoted in a single article.