I actually thought polars’ lazy api would allow for out-of-ram computation?
Also dask is more flexible than spark, since it lets you deal with numpy arrays and arbitrary objects better than spark can.
Also dask is more flexible than spark, since it lets you deal with numpy arrays and arbitrary objects better than spark can.
Calling `collect(streaming=True)` on a `LazyFrame` will allow you to process datasets that don't fit into memory. This currently works for groupbys, joins, many functions, filter etc.
We will extend this to sorts and likely other operations as well.
I know its a boring use case, but the challenge with it is that it is a complete waste of money and carbon footprint to use Spark to process a 20 MB CSV or table with few thousand records, but tools like Pandas fall apart when you hit a 50 GB CSV or table with few billion records.
Something more efficient (say, in Rust and not Python or Java) and yet scalable (due to not fitting everything into memory) would be a great help here.
And we will extend functionality for out of core processing. A single node can do a lot!
dask.bag has generic parallel processing capabilities. Query a database, a rest api, something. Then merge into dataframes across dask workers.