Databricks/Spark is overpriced garbage for Python-toting data scientists/engineers/analysts who don't know how to model data or write SQL. Exactly like Hadoop was 15 years ago for the Java-toting data scientists/engineers.
Databricks is their hosted version just like Aurora is a hosted PostgreSQL.
The fact you don't seem to understand the difference makes me question if you are qualified to be making statements that it is garbage and we can simply write SQL. Because having worked on many large, big data projects SQL is often not the right tool for every use case.
Some evaluation Criteria:
- Ease of maintenance and operation is almost paramount.
- It's fine if the solution never lives anywhere but 1 single virtual server that scales vertically (data might grow to a couple TB, but not PETA BYTES)
- Similarly, 20 9's is not a criteria. If the machine fails and it takes an hour till someone goes and re-deploys, that's fine.
- Declarative, reproducible deployment with an easy upgrade story would be great
- Ideal if the deployment can be run locally for quick developmnet