Anyone using non-JVM stuff in general? Nothing against the giant ecosystem around Hadoop/spark etc.

I am currently using one Python process with Prefect for DAGs (https://docs.prefect.io/guide/), custom API queries and Elasticsearch indexing code in a batch processing style and it seems to be going fine to ingest a 2TB dataset with the ability to generate rich errors, retry/resume etc.

Besides maybe job listings, why would a real "plumber" who came from python and js NEED to dive into the whole Apache ecosystem?