marsupialtail_2

None of the benchmarks involved any UDFs.

Polars is adding that feature itself. You probably just have to wait a couple weeks

I have. I am about 2x faster than trino with fault tolerance. But I didn't put the numbers on that plot because this trino feature is still really new and I might not be benchmarking it in the best way.

I only wrote Python. I am using C++ and Rust libraries

Yeah it's just that the UDFs I see people need require Python packages like sklearn and PyTorch that really don't have good alternatives in other libraries.

that's certainly one way to look at it

Yes -- we are building a sql compiler to the dataframe API. It currently passes half of TPC-H, but is not really ready. It will be open-sourced soon.

The SQL optimizations like predicate pushdown and early projection are all there already in the dataframe API, similar to Polars.

Name?

The focus is olap and etl not oltp. In fact I assume data doesn't change :-)

SQLGlot is really great, check it out!

I don't really compile python functions to sql -- I just execute sql. It's just that the execution engine is written in python

hmm I wasn't aware of https://github.com/apache/arrow-datafusion-python... thanks for the pointer.

time series target release by April this year. main challenge is supporting them in the SQL API -- execution engine support is already done

Firstly quokka is pure Python, which I believe is good for interoperability with Python based UDFs.

Secondly quokka tries to be fault tolerant. I.e. it can handle worker failures intra query and not have to start over. This is quite important in real world spark deployments with thousands of nodes running many hours. AFAIK this is not well supported by most spark alternatives.

Finally quokka has a much stronger focus on time series data analytics. It is meant to excel at workloads like range joins and asof/PIT joins used for feature engineering. (This part isn't too stable yet so is not open source)

This means quokka optimizes on a different point in the UDF/performance/fault tolerance tradeoff space than something like arrow ballista or starrocks, which I think are pure performance plays

Checkpointing is actually a bit difficult for fault tolerance for long running batch computations and state tends to grow linearly as the computation progresses, unlike streaming applications where you typically use windows. This means periodic checkpointing leads to quadratic amount of writes to durable storage which is pretty bad.

Quokka supports checkpointing but does not enable that by default to prevent this common problem from killing normal operation performance.

Perhaps you can write the underlying query engine :-) in something that's not Python lol

I just find it funny how my post on why I wrote a SQL engine in pure Python is right next to this one on the front page

Ask me anything :-)