So Databricks is being replaced by open source, self-managed spark-on-k8s? Broadly speaking? Any concrete recommendations on how to do that in a project?
We tried this at a previous company I was at, it was incredibly tough to maintain and tune. We ended up using databricks which made it significantly more productive.
No, it's not about the technology. It's about your stack and skill set. If you are still on the school of one box for all. Then just pay for Databrick or EMR. If you are already running K8s because you invested into that world, then adding Spark isn't a huge jump.