Hacker News Reader

Comment by 0thgen | Hacker News Reader

Ive found polars quite intuitive, though for python, I lean more towards [ibis](https://ibis-project.org/). The interface is nearly identical, but ibis has the benefit if building sql queries before pulling any actual data (like dbplyr) — whereas polars requires the data to be in-memory (at least for rdb’s, though correct me if Im wrong)

this to me seems like a good argument for only using ibis, but Im happy to be convinced otherwise

View on HN

I haven't used it but polars has a lazy api https://docs.pola.rs/user-guide/lazy/using/#using-the-lazy-a...

Polars doesn't require all data to be in memory. It has a lazy API, optimizations that prune data at the scan level and parts of the engine can process data in batches.

Algorithms like joins, group bys, distinct, etc, are designed for out-of-core processing and can spill to disk if available RAM is low.

If it all fits in memory, isn't it simpler to just load it first? Sure, you're kinda painting yourself into a corner, but it can be a big corner. e.g. I know that until recently Amazon still ran a bunch of key recommendation algorithms on a single big machine each night, because it's just simpler to load everything into memory and crank on it.

When reading from the dataframe, depending on the columns, filters, and aggregates you want returned, most query engines can skip data that is irrelevant, making the query run much faster.

In your amazon example the data is probably optimized for fast lookups on a small number of records, like an in-memory cache.

Simpler, but you can avoid huge amounts of work just simply by not loading what you need. That's the beauty of the scan_* apis in polars. Most query engines have some level of support for io skipping.

What backend do you use? The little spin I've given Ibis felt like it's not deeply supported with a lot of the backends.

Reply on news.ycombinator.com