Our experience with Spark has been horrendous. Very unstable. Marginal improvements. Big hassle.
Would strongly advise you to consider hadoop. We also used storm and found it to be much stable.
Databricks makes a lot of noise though.
Would strongly advise you to consider hadoop. We also used storm and found it to be much stable.
Databricks makes a lot of noise though.
I am partly asking this because you clearly feel strongly about this topic (your account was created an hour ago, most likely to comment on this).
Spark on YARN has been very stable and zero hassle for us. Run one command and it is deployed around the cluster and up and running. For performance Spark absolutely destroys stock MapReduce/Tez especially if like us you have a cluster with lots of basically unused RAM.
And Spark SQL, Spark Shell are both fantastic additions to the Hadoop ecosystem.