I wonder if similar performance can be achieved with Spark accelerator like https://github.com/apache/datafusion-comet. Of course it didn’t exist 4 years ago, but would it cheaper to build?
Ray allowed them to optimize elements that spark didn’t, and that was what improved performance, not that spark itself was slow.