Hacker News Reader

Comment by data-ottawa | Hacker News Reader

I've been using pandas for over a decade and I'm very efficient with it.

The Modern Pandas blog series from Tom Augsburger is excellent, and the old Wes McKinney book (creator of pandas) gave a good glimpse into how to effectively use it.

I don't think many people learn it correctly so it gets used in a very inefficient and spaghetti script style.

It's definitely a tool that's showing it's age, but it's still a very effective swiss army knife for data processing.

I am glad we're getting better alternatives, Hadley Wickams tidyverse really showed what data processing can be.

View on HN

I also recommend Matt Harrison's talks ("Idiomatic Pandas") and book ("Effective Pandas").

Once I adopted method chaining, a lot of the issues that I had with pandas in the past due to poor style (e.g. SettingWithCopyWarning) pretty much disappeared.

Just imagine how much more efficient you would be if you were using R's DataTable.

Look, I applaud your skill, but at some point even a master craftsman realizes that the swiss army knife may not be the best tool, and a leatherman offers certain advantages.

Sometimes I write data analysis code in R to make myself aware of how it is occasionally possible to have nice things.

I really like R’s library and I’ll use them any chance I get (libraries like lmer are still orders of magnitude more efficient than the same model in Statsmodels).

From my experience the biggest impediment to using R in production is many orgs don’t have a blessed way to run it.

R is my favourite language for data processing, the manual section Computing on the Language[1]is why R is such an ergonomic tool. I had hoped Julia would catch up, but Julia’s macros are not comparable in their depth.

I think pandas is probably the data equivalent of editing files using default vim or processing data with awk.

[1] https://rstudio.github.io/r-manuals/r-lang/Computing-on-the-...

As a joke, I wrote an Ibis backend (https://github.com/cpcloud/ibish) that processes expressions using shell commands strung together with named pipes. It supports joins using the coreutils join command, projections, filters and some aggregations with awk.

It's faster than pandas in some cases and folks should put it into production immediately!

fun fact if you weren't aware, Ibis was also originally created by Wes McKinney and takes a lot of inspiration from the tidyverse

I was not aware of that, the influence from the tidyverse is very clear

I’ve tried to make mockups of similarly inspired API and I think they did a great job with clever tricks like the `_` reference (does that clash with IPython/Jupyter’s use?).

I do wonder how they manage compatibility with so many backends, it seems like many features will be directly tied to your backend (e.g. Trino using Java Pattern for regex vs BigQuery using Re2) in hard to explain ways. But maybe that’s not a big concern, because very often you’re only going to be using a couple of backends.

I’ll have to try Ibis out for myself, it looks like it can unify a lot of the work I have to do. I’ve moved away from pandas for all but the last mile of computation, so this might be a direct good SQL replacement.

`_` definitely clashes with IPython and Jupyter's use of it.

However, if you import it from Ibis then it ceases to be used as "most recent result" and remains the ibis underscore object (unless of course you explicitly assign it to something else).

Regarding backend compatibility, there are definitely a few kinds of things that we don't currently abstract over. One is regular expression syntax and another is floating point math (e.g., various algebraic properties that are violated that result in slightly different outputs).

Hope you give it a go, and please report issues at https://github.com/ibis-project/ibis.

Thanks for the reply!

I think that’s the right policy to take, and I did notice the support matrix on your website which addresses my earlier question:

https://ibis-project.org/backends/support/matrix

Reply on news.ycombinator.com