The OP is the original creator of Ballista, so he's well aware of the project.
Ballista is much less mature than Spark and needs a lot of work. It's awesome they're making Spark faster with Comet.
Ballista is much less mature than Spark and needs a lot of work. It's awesome they're making Spark faster with Comet.
The Comet approach is much more pragmatic because we just add support for more operators and expressions over time and fall back to Spark for anything that is not supported yet.
I don't think it's a fair criticism of Ballista to say that it failed in any way. It just looks to need substantial effort to bring it on par with Spark. The performance benefits are meaningful. Ballista can then not only take the crown from Spark, but also revalidate Rust as a language.
I do see a new opportunity for Ballista. By leveraging all of the Spark-compatible operators and expressions being built in Comet, it would be able to support a wider range of queries much more quickly.
Ballista already uses protobuf for sending plans to executors and Comet accepts protobuf plans (in a similar, but different format).
We are in the enterprise with large cloud budgets and can simply change instance types. If you're 20x then that is a different story but then (a) you need to have feature parity and (b) need support from cloud vendors which Spark has.