Hacker News Reader

Pandas doesn't. Polars I think has some lazy-loading capability, but it's not the default mode of operation and I don't think it supports all features. If DuckDB doesn't, then that's a big advantage.

View on HN

I think you mean that Pandas does require your entire data to fit in memory? https://pandas.pydata.org/docs/user_guide/scale.html

Yes, thank you, that was a typo.

Apparently duckdb paired with arrow lets you work performatively on bigger than memory parquet data (I haven't tried this though)

Arrow seems to be the common denominator for larger than ram. Polars is completely built on arrow, so the same advantages should apply.

https://github.com/pola-rs/polars#handles-larger-than-ram-da...

Reply on news.ycombinator.com