Comment by liminal | Hacker News Reader

Comment by liminal | Hacker News Reader

Parent Full thread

This looks amazing.

* I started poking around the docs and I'm most excited about a Ray backend runner. I'm hoping this allows more ergonomic distributed data frame computation on an existing Ray cluster.

* Is this based on Apache Arrow? I would assume so, but it's important that it be zero-copy from other tools. Would like to see this mentioned prominently somewhere.

* I really like the Polars expression API. I haven't dug into the docs enough to know how it relates. I do see a reference to pinning the embedded Polars version, so fingers crossed there's compatibility. It would be AMAZING to be able to take Polars code and run it in a distributed cluster with minimal changes. Can anyone chime in?

> Ray backend runner

Yes, give it a whirl and let us know what you think! Ray is amazing and has actually gotten a lot better post their 2.0 release :)

> Is this based on Apache Arrow?

Indeed it is, and thanks for the feedback. We'll make this a little more visible. We use the arrow2 Rust crate (same one that Polars uses) for our in-memory data representation.

Our data representation makes it such that converting Daft into a Ray dataset (`df.to_ray_dataset()`) is actually zero-copy. So you can go from data transformations into downstream ML stuff in Ray really easily.

> It would be AMAZING to be able to take Polars code and run it in a distributed cluster with minimal changes.

Unfortunately we don't have Polars API compatibility. This seems to be a recurring theme in this thread though. The problem is that certain Polars expressions are non-trivial to do in a distributed setting, and Polars itself as a project is so young and moves so quickly it's hard for us to maintain 100% API-compatibility.

That being said, you are correct that a lot of the API is very much inspired by Polars, which should hopefully make it easy to move between the two.

Please at least reach out to the Polars folks and see what's possible.

Reply on news.ycombinator.com