- Much faster and more memory efficient.
- Consistent Expression and SQL-like API.
- Lazy execution mode, where queries are compiled and optimized before running.
- Sane NaN and null handling.
- Much faster.
- Much more memory efficient.
- Consistent Expression and SQL-like API.
- Lazy execution mode, where queries are compiled and optimized before running.
- Sane NaN and null handling.
- Much faster.
- Much more memory efficient.
I dont care so much about the memory and CPU stuff, I mostly leave the heavy lifting to an SQL engine.
Although the Null handling seems very compelling, I guess it comes at a cost of incompatibility with existing libraries, otherwise Pandas would have implemented it as well?
I am curious about the SQL api though.
If it were better, we'd use it internally in Ibis for the Polars backend implementation.
If you're going down the mixed SQL, DataFrame API route then Ibis is probably the best solution out there for that.
I work on Ibis, so take what I say with a grain of salt. There may yet be other libraries out that there that have similar functionality.
And by now I know that very well.
Like someone-screaming-in-my-ears-know.
I am starting to think that Polars is showing all the signs of a hype or a cult.
I am still not convinced, particularly since the community feels more like a marketing department than someone who wants to genuinely help.
I can do that thing you describe with SQL.
If you mean whether I run it distributedly a la Spark then no. If you mean whether I test it on various machines with different RAM sizes then yes.
> I dont care so much about the memory and CPU stuff, I mostly leave the heavy lifting to an SQL engine.
Well, I care. Both pandas and polars are, to my view, single-machine dataframe library, so the memory and CPU constraints are rather stringent.
My comparison is based solely on my experience: reading csv files that are 20% to 50% the size of RAM, pandas takes (or errors out after) 2 to 10 minutes, while polars finishes in 20 seconds. Queries in pandas are almost always slower than polars.
But reading your comment, it seems you and I have different use cases for dataframe libraries, which is fine. I mostly use them for exploratory analysis, so the SQL api is not that much of a plus to me, but the performance is.
Many cloud providers now offer serverless SQL and Spark capacities (serverless=no set up for you). This is the magnitude change for me.
With pandas you can maybe process 10 million rows, with polars maybe 50 million. But with a distributed service maybe 100 times more?
I'd suggest less snark, you're not doing yourself any favors.
Why do people feel the need to jump in and police tone like this? Who are you? You're not doing yourself any favours, either.
>common literary technique to emphasize a point.
"Common" is a stetch, and who cares.
From the community guidelines(https://news.ycombinator.com/newsguidelines.html): "Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes."
> Who are you?
A member of the community.
Did you know that more information is communicated via tone than words?
The original purpose of my comment was to, in a light-hearted way, point out redundant information, not to incite a flame war about tone on HN.
This thread reminds me of why I don't come here very often.