Hacker News Reader

Not an answer, just wondering if anyone is using Apache Spark for ETL?

Is this approach viable or if it is just marketing:

https://databricks.com/blog/2017/01/19/real-time-streaming-e...

I use spark for batch data transforms, our data is structurally complicated, but not for streaming. I attended the EU Spark Summit last year, as of then Spark streaming wasn't prime-time ready, I don't know if anything has changed in the last 12 months on the streaming front.

You should have a look at Flink if you're looking into streaming transforms.

Reply on news.ycombinator.com