Spark being an example of a tool that (also) exists inside the Hadoop ecosystem and is still gaining acceptance due to popularity, but at the same time losing traction as better option are arising.
What better options? Can you drop some links? Thanks :)
Depending on cloud vs on-prem.
Not really, just use Apache Beam and set the runner to whatever you have / prefer
No thanks. I do not see any value of the n+1 unnecessary abstraction over things that I am already familiar with. An average customer does not want to use these things either.
So the main issue with Spark is that the streaming is not that great.
In general a Spark cluster persé is not bad, but I see more and more people using Apache Beam and then just use a runner that either they already have or that fits them best. https://beam.apache.org/