Good riddance.
Pandas turns 10x programmers into 1x programmers.
Pandas turns 10x programmers into 1x programmers.
The Modern Pandas blog series from Tom Augsburger is excellent, and the old Wes McKinney book (creator of pandas) gave a good glimpse into how to effectively use it.
I don't think many people learn it correctly so it gets used in a very inefficient and spaghetti script style.
It's definitely a tool that's showing it's age, but it's still a very effective swiss army knife for data processing.
I am glad we're getting better alternatives, Hadley Wickams tidyverse really showed what data processing can be.
Once I adopted method chaining, a lot of the issues that I had with pandas in the past due to poor style (e.g. SettingWithCopyWarning) pretty much disappeared.
Look, I applaud your skill, but at some point even a master craftsman realizes that the swiss army knife may not be the best tool, and a leatherman offers certain advantages.
From my experience the biggest impediment to using R in production is many orgs don’t have a blessed way to run it.
R is my favourite language for data processing, the manual section Computing on the Language[1]is why R is such an ergonomic tool. I had hoped Julia would catch up, but Julia’s macros are not comparable in their depth.
I think pandas is probably the data equivalent of editing files using default vim or processing data with awk.
[1] https://rstudio.github.io/r-manuals/r-lang/Computing-on-the-...
It's faster than pandas in some cases and folks should put it into production immediately!
I’ve tried to make mockups of similarly inspired API and I think they did a great job with clever tricks like the `_` reference (does that clash with IPython/Jupyter’s use?).
I do wonder how they manage compatibility with so many backends, it seems like many features will be directly tied to your backend (e.g. Trino using Java Pattern for regex vs BigQuery using Re2) in hard to explain ways. But maybe that’s not a big concern, because very often you’re only going to be using a couple of backends.
I’ll have to try Ibis out for myself, it looks like it can unify a lot of the work I have to do. I’ve moved away from pandas for all but the last mile of computation, so this might be a direct good SQL replacement.
However, if you import it from Ibis then it ceases to be used as "most recent result" and remains the ibis underscore object (unless of course you explicitly assign it to something else).
Regarding backend compatibility, there are definitely a few kinds of things that we don't currently abstract over. One is regular expression syntax and another is floating point math (e.g., various algebraic properties that are violated that result in slightly different outputs).
Hope you give it a go, and please report issues at https://github.com/ibis-project/ibis.
I think that’s the right policy to take, and I did notice the support matrix on your website which addresses my earlier question: