Hacker News Reader

Comment by sakras | Hacker News Reader

I think the most interesting part of this project is the fault tolerance. I can’t say I’ve seen any other projects do this, but it seems reasonable to want checkpointing during a long computation.

Another thing I like is that conceptually it seems like it would be simple to switch the underlying query engine (right now it’s Polars) in the future. Seems like a pretty general distributed system.

View on HN

Checkpointing is actually a bit difficult for fault tolerance for long running batch computations and state tends to grow linearly as the computation progresses, unlike streaming applications where you typically use windows. This means periodic checkpointing leads to quadratic amount of writes to durable storage which is pretty bad.

Quokka supports checkpointing but does not enable that by default to prevent this common problem from killing normal operation performance.

Fault tolerance is an important feature but, given Python’s popularity in data science, the most interesting part of this project to me is support for Python UDFs.

In principle, the programming language should not be the greatest consideration because developers can learn and use different languages for different applications. In practice, being able to draw on familiar syntax and libraries can make a real difference in usability.

Perhaps you can write the underlying query engine :-) in something that's not Python lol

Even the self driving in Tesla use Python. If your Python is slow, you are using Python wrong. So it’s kind of a good thing that it’s natively slow, because you are not supposed to do your computations with native Python. It it was faster (added a lot of type hints and build processes), you would probably not be nudged in the right direction.

https://cerfacs.fr/coop/fortran-vs-python

Self driving for TSLA cars is not really the benchmark I'd use for "something that works reliably"

I don’t think there’s any evidence that FSD’s reliability is affected by being written in Python.

Polars is already not python

Reply on news.ycombinator.com