I'm amazed that the implementation was under 1500 LOC! Was that the research prototype or the shipped preview?
Congratulations on the VLDB paper! Hopefully I'll come say "hi" in LA :)
The shipped preview has only a bit more than 1500LOC.
The VLDB paper was presented at Rio in Aug this year already, but I'll try to come over to LA anyways :)
- https://medium.com/teads-engineering/spark-performance-tunin... - https://www.inovex.de/blog/efficient-udafs-with-pyspark/
There are definitely some differences between the kind of UDFs that Spark supports and the kind that Froid handles. For one, Spark UDFs cannot invoke a Spark SQL query in their definition AFAIK, whereas TSQL functions can. But still, some techniques might be applicable. Definitely worth digging further!