So where does this place Polars? My perhaps incorrect understanding was this (Arrow integration) was a key differentiator of Polars vs Pandas.
Some other key differentiatiors:
- multi-threaded: almost all operations are multi-threaded and share a single threadpool that has low contention (not multiprocessing!). Polars often is able to completely saturate all your CPU cores with useful work.
- out-of-core: polars can process datasets much larger than RAM.
- lazy: polars optimizes your queries and materializes much less data.
- completely written in rust: polars controls every performance critical operation and doesn't have to defer to third parties, this allows it to have tight control over performance and memory.
- zero-required dependencies. This greatly reduces latency. A pandas import takes >500ms, a polars import ~70/80ms.
- declarative and strict API: polars doesn't adhere to the pandas API because we think it is suboptimal for a performant OLAP library.
Polars will remain a much faster and more memory efficient alternative.