You're probably going to be hard pressed to really beat out the sheer data throughput capacity of BigQuery without significant developer investment. I've done a lot of Apache Spark, Flume and Beam and they all have their own quirks and problems points. All of them are not at interactive speeds, have OOM issues for certain operations and require a lot of glue code/data management.

Honestly at this point you're probably going to want to start looking into the query planner and execution stats of your BigQuery jobs to see if you're doing something really weird and optimize that: https://cloud.google.com/bigquery/docs/query-plan-explanatio...