Pandas Should Go Extinct | Hacker News Reader

Pandas Should Go Extinct

eddie.codes

194 points__eddie__101 comments

It's been a while since I've seen an actual data science post submitted to Hacker News: both because AI has superset a lot of DS tasks (e.g. vector embeddings), but also because not much new has happened in DS. Polars has been around for a bit and as noted it is much better than pandas, but otherwise the DS ecosystem has been somewhat stagnant.

I'd write more tutorials about how to use data science tooling but one consequence of AI is that all the old data sources I used to analyze such as social media and Reddit are now completely locked down (I am surprised NYC Taxi is still being updated, though). Therefore in the meantime, I'm working on making better data science tooling...although unclear to what end due to the data issue above.

I believe the flood of AI articles drown the fact that data science is not dead.

I see a lot of work towards Geo-Data Science.

PostGIS was a first generation. Now we have GeoPandas, DuckDB Geoanalysis tools and Sedona DB.

IMDB data set is also up.

What do you mean? Data science is advancing at an incredible pace IMO! DuckDB is the coolest thing to happen to data science in my neck of the woods

Cute title; I thought I was about to read a contrarian ecologist saying "a species that is so very specialized in its diet and finicky in its reproductive behavior doesn't deserve, evolutionarily speaking, to survive". (Seriously, it's seriously freakin' difficult to get pandas to reproduce in captivity). And while my knee-jerk reaction would be "maybe, but we should still preserve them because we can", I was prepared to see if the author had a serious argument to present.

Instead it's a cute bait-and-switch title, and the article tells you upfront that it's actually about the Python `pandas` library. Which I think I've encountered maybe once in my entire career (I'm not in the data-science field), so I don't have much meaningful to say about the article itself. I just want to commend the author on fooling me with the title. This is the kind of "clickbait" I can respect and actually wish there was a little bit more of sometimes. A nice chuckle, then a real article.

Yeah I wanted the controversial panda article. Can someone write that instead?

Apologies for not rewarding your trust ;)

Would be very interested to read an article on the literal interpretation of my title.

Also just wanted to say thank you for this comment - made my day :)

> Seriously, it's seriously freakin' difficult to get pandas to reproduce in captivity

It is impossible, and has been tried many times by many people, to reproduce Octopus vulgari in captivity. Their reproductive behaviour looks also quite weird for us (the mother always die). Yet they thrive in the wild unless over fished.

because breeding intelligent creatures generally requires keeping them happy. When treated like a farm aniaml, many high cognition anaimals will refuse to reproduce, just look at how bad life has become for average humans, and predictably, humans are starting to refuse to reproduce.

Birth rates fall as prosperity increases.

Whatever this is, it's not prosperity.

What other countries have you visited?

GDP doesn't measure prosperity.

Prosperity doesn't measure happiness.

to replacement, we're rapidly falling below. distributuon and related trends are extremely grim.

That's more true in some locations than in others. There are places where the reported quality of life is quite a bit higher than others. I see comments like yours a lot on HN, and I'm starting to suspect that the average HN commenter is more likely to be living in some of the poorly-run big cities than in the better-run small-to-medium-sized towns. (Specifics can, and will vary. But on average I've found I'm much happier living in a medium-sized town than in a megacity, and I've also found the city governments of medium-sized towns to be more likely to be competent. Perhaps I've just been lucky in the medium towns I've lived in, though; I'd need far more data before I was certain of the thesis).

Point is, if you're miserable in the place where you are, consider whether moving to a different town is feasible for you. It's not always an option for many people, for many varied reasons. But it's definitely something to consider. If it turns out you can do it, sometimes taking that risk can pay off in the long run.

yes? im not sure what your point is, i sense you seem to disagree, but what youve said supports what ive said so im not sure how to engage.

My point was that "captivity reproduction" should not be a key point in the viability of a specie. Octopus can live in captivity (albeit stressed!), but for some reason their larva don't survive. Do they deserve to go extinct for that? They were able to survive for 500 million years, through some massive extinctions, looks good enough to me.

There is indeed a whole book about ecosystems and charismatic megafauna called "Do We Need Pandas?: The Uncomfortable Truth About Biodiversity"[1]. And there is debate among naturalists [2]

[1] https://www.goodreads.com/book/show/9656950-do-we-need-panda... [2] https://www.theguardian.com/environment/2009/sep/23/panda-ex...

I was expecting a diatribe about the (UK) police: "Next week, right, I'm going to blow up a panda in Croydon".

> People typically start with Excel and graduate to Pandas somewhere in the GB range. Pandas serves them well into the 10s of GBs range, and then they start hitting memory issues, slow computation, or become frustrated with Pandas’ baroque API.

Assumes that a project moves beyond 10s of GBs. I guess 99.9% of projects that import pandas fall well below this threshold.

At my work I had convinced the ML pipeline engineers to switch from pandas to polars for even small ETL pipelines and there were notable performance gain with better CPU/memory utilization.

If a library is performant at large datasets, it is likely performant at small ones too.

I’m not disagreeing with that statement at all. You missed my point that there are thousands of people making small Python scripts for education and personal projects everyday. In those circumstances the performance concerns are irrelevant and the ergonomics of good pandas documentation and community knowledge make it a better choice.

That inertia is not a good thing, and it's partially why there's stagnation in data science. Polars is more than mature enough in both documentation and resources for it to be a daily driver.

True, but also if Polars and DuckDB offer a similar experience with the ability to scale beyond that range, why not use them (for new projects)?

This isn't a call to arms to rewrite everything in the new shiny, just consider the new shiny for new shiny things

Not to mention less footguns. I used to spend my days unwinding bad habits DS pick up from years of panda abuse.

Credit to pandas for popularizing dataframes in Python, but polars and duckdb are objectively better APIs in addition to their implementation improvements. Agree that it’s time we let it go.

Yes, this was something I emphasised more in the slides / my actual talk. But it's just significantly easier to reason about the APIs for both DuckDB and Polars

Most folks just need to learn how to use pandas well and that will open enough doors. Then they can move to polars or duck if needed.

People are usually surprised to hear that polars can be slower for fairly pedestrian operations, especially with smaller datasets. For example take a 5000 x 3 dataframe of float64 and sort by one column and you'll find polars takes about 2.5x as long. If you set POLARS_MAX_THREADS to 1 then it's faster. Though this all depends on the machine. Polars tends to shine with larger datasets or where it can heavily take advantage of query planning.

Don't skip your profiling

(Without profiling or looking into this at all) I'd guess this has to with thread creation, inter-core communication/latency, and possibly having to merge results or otherwise interleave operations. SMT is another likely candidate.

Regardless, CPUs are really good at single-thread.

In my opinion a better argument to stop using pandas is the very unintuitive API pandas have. Additionally, a slight change in the query can force you to restructure the whole query (change all lines), while in Polars (and tidyverse in R) it's just a simple one-line change.

If you learn to write pandas correctly you end up writing it very similar to polars (or tidyverse).

I agree that the API has warts, though typically it is more concise than polars.

The problem is a small change in the question can force you to make huge changes in the code in pandas. I recently gave some examples in my blog [1]. E.g. compare the last two code blocks, where the small change is just that the median is taken within countries. This requires several line changes in pandas.

[1]https://bjarkehautop.github.io/Website/blog/data-wrangling-t...

Personally, I think the inclusion of lambda functions in the pandas code shows that the approach isnt ideal to begin with. Fine on smaller datasets, but typically becomes a slow and cumbersome overhead when interacting with larger datasets. More verbose per line vector operations are generally orders of magnitude faster, and become more so as the data grows. They also tend to be easier to read / identify what is going on or what the intent is for people unfamiliar with the code base.

This is a common complaint I get all the time (heard it this week while teaching pandas). I compare this to whitespace indentation in Python.

Lots of folks complain about it before using it. After they use it it is a non issue.

If it really is an issue (and it generally isn't a cause of vectorization removal when used correctly) and you can't get over the syntactic noise if the lambda, pandas 3 introduced pd.col (that work in most (I filed a big about some exceptions) places when you'd use lambda).

Ive seen it in such a wide variety of scenarios where it causes vectorisation removal because of its implementation that I dont reach for it by default any more. I've a number of devs around me who have similar opinions. Each know how to use it, but you dont always write the code you're reviewing or optimising.

Would love to see examples.

I looked at your code very quickly, but it looks like you need to use .filter after a .groupby...

The text right above the code says why you can't...

edit:

Let me clarify. From the blog-post:

> since a `DataFrameGroupBy` object doesn’t have a `.query()` or boolean-indexing shortcut of its own, so filtering within groups needs `.apply()` again, and the surrounding pipeline has to be rebuilt around it:

Hence you really do need one of the versions of the code I gave. You can't do the naive approach with just `.groupby().filter(lambda: )`, since you need a row-wise decision.

I'm confused, you can use filter after a groupby in pandas...

It's late here, I'm going to bed, perhaps I'll write the code tomorrow when I'm at my laptop and not on my phone.

I misspoke, you need to use .groupby/.transform to add a new filtering column:

    (sales
      .assign(country_median=lambda df_: (
          df_.groupby("country")["amount"].transform("median")
      ))
      .query("amount <= country_median * 10")
      .assign(net=pd.col('amount') - pd.col('discount'))
      .groupby("country", as_index=False)
      .agg(total=("net", "sum"))
    )

Yes, so basically equivalent to the code I showed in the blog.

And for a similar dplyr-like API on duckdb I recommand the excellent https://ibis-project.org/ !

Am I crazy or did the OP swap the contents of the posts around accidentally?

https://eddie.codes/posts/pandas-should-go-extinct/ <=> https://eddie.codes/posts/source-code-comments/

Yep. Was very confused about the title and URL slug; reached the end waiting for Pandas to be mentioned.

Sorry, fixed it now!

Published two posts at the same time and total PEBCAK

No worries, we all make mistakes, at least some one is thinking of the bamboo: https://www.youtube.com/watch?v=WqX2aqXbwB0

I'm in the middle of wrapping up the edits for Effective Pandas 3rd Edition. (I also wrote a Polars book and just wrapped up a weeklong training session on pandas this week.)

Pandas is not perfect, it has a bunch of warts. But it is good enough for most. (And many of those folks are using Excel or tableau or power bi... These were the types I was training this week).

If you have medium data, migrating from pyarrow backed pandas to duck or Polars is trivial.

Out of curiousity, why wouldn't someone just migrate directly to Polars?

I use polars or duckdb now exclusively. Better syntax, better performance. But pandas is deeply entrenched - I try to get my team off it but it’s an uphill battle. It’s not going anywhere anytime soon.

Only in the last few years did I start using SQL properly. Before that my pipelines would live in python. Now I offload as much to the db as possible, and keep my python simple glue. I'm very happy with this compared to other methods in pandas or polars.

If I still need to do db-like things in python I think duckdb is better.

Sup Eddie, the actual motivating example is we finally nixed Pandas from Data Ingestion, now reading excel files takes 2 seconds instead of 2 minutes. However unfortunately your

> TODO: rewrite this entire service

remains.

> Sup Eddie, the actual motivating example is we finally nixed Pandas from Data Ingestion, now reading excel files takes 2 seconds instead of 2 minutes. However unfortunately your

I shall pass this information along to my sleep paralysis demons. They'll be glad to hear it.

> > TODO: rewrite this entire service

Part of me enjoys the historical significance of this. However, I feel like keeping it would violate my own princples: https://eddie.codes/posts/source-code-comments/

Rewriting pandas to polars is relatively trivial for most tasks these days. Especially if you wrote your pandas code correctly.

I still prefer (and use) Pandas for EDA. I think matplotlib integration is a better choice for most viz.

Also, I'm probably in the top 3-5 worldwide for number of folks I've trained with pandas. I offer Polars training and there is little demand for it.

Besides the performance benefits, I use Polars at work because it’s just (subjectively) nicer to work with. The “pl.col” API lets you create arbitrary generated/virtual columns anywhere you want, declaratively. You can throw in these column expressions in wherever without actually computing their values and storing that in memory. Very powerful stuff.

Polars seems nice but in my experience using it, the "lazy" APIs would still immediately materialize a ton of stuff in memory and had very spotty support on what data formats and storage integrations were possible with scan_* functions (though that was half a year ago and the support is slowly improving). It's frustrating, I mean really frustrating, to think I could solve a lot of my "scan through heinous amounts of data without any memory hungry things like window aggregations without blowing out my memory" with Polars and then watch my scan_thisorthat() call result in instant memory usage ballooning.

DuckDB on the other hand is wonderful and truly doesn't use any more memory than it really needs to.

Did you ever report these issues to polars developers?

I reported many bugs to both pandas and polars over the years and both teams tend to address relatively big ones. (Still have some outstanding pandas bugs that I think are a big deal but the devs disagree.)

Yeah, was in the Discord for a while

That sounds like there's something going wrong, because that definitely shouldn't be the case! Although it sound like you've given up on it: if you have any example to send me in the right direction I can look into that for you. I'd love to know if there's something I can relay to the team.

Thijs, DevRel @ Polars

> If you were interested in whether cash usage declined during the Pandemic, it sure did. However, correlation !== causality, so please don’t draw any meaningful conclusions from this.

I think we can be certain that reduced cash use didn't cause the pandemic, and a common cause for both is also hard to imagine. That only leaves the pandemic causing a reduction in cash use. Am I missing something?

Yes. Both having different causes, just at similar times.

Claude loved pandas, at least around 4.5 opus. I had to keep reminding it to use polars instead. I wonder if it’s changed since.

You could put in your AGENTS.md/CLAUDE.md for it to use polars.

Modern LLMs use it just fine.

These days it just writes its own DataFrame library for you

Makes sense, especially with AI coding tools the rewrite and familiarity arguments hold less water. Similar for the rustify everything crazy.

The problem is, orgs who see themselves as big data orgs want to act that way, even if they're medium data. "But we'll need it when we grow", "we need to know the state of the art tools"

The reality is most of the pandas audience don't care about performance. Whenever performance is in the question people have always used polars, duckdb, dask etc. These users are usually software engineers and not data analysts. Agreed, that there is a real gap in latency and performance though!

Nice post but they quickly disregard Dask, don't explain why, don't test it, even exclude it from the benchmark they quote. I don't know if is the better answer, but it seems worth testing if you want a balance approachable + scalable. That's kind the thing Dask was meant to do.

We run Dask distributed for contents in what their chart covers (10TB+ with 4+ physical hosts) and it's a problem that can't really be discussed without well defined drive read or network latency limitations (if non-local storage). We've found defining recordsize, raid and other filesystem patterns to have an equal or larger impact in itself. Pandas is great because it allows us to validate on a sampling or smaller time window dataset, and then the same engineers can use a very similar implementation for Dask distributed through their scheduler tool.

While throwing hardware at problems isn't always a reasonable solution, it's trivialized to discuss this without IOPS or other read limitations.. and the nvme enterprise market is rapidly developing making this change quickly.

I've been saying this since using Databricks at a company almost a decade ago. Most folks do not need big data tools, and it's just so entrenched because everyone wanted to be a "big data" company and pandas was how you handled big data.

Real link here: https://eddie.codes/posts/source-code-comments/

Something going wonky on their blog, where two posts got their links swapped.

Yep sorry, fixing now. It's what you get for being bitten by the bug to write twice in the same day

Fixed!

So it wasn’t just then, for me it linked to their “Useful code comments” post

Liked the title

I'd drop pandas if polars worked eamlessly with sklearn.

You can do to_pandas() and from_pandas() which doesn't have much overhead, or to_numpy() directly which is typically zero-copy and thus no overhead: https://docs.pola.rs/api/python/stable/reference/dataframe/a...

Most of the time it works, but there are still corner cases throughout the python ML ecosystem where polars fails.

This and the decision not to natively plot with matplotlib keep me in pandas for most tasks. (Plus there's is still a relatively large demand for pandas training.)

The article is harsh for no reason. Pandas served us, for a lot of time and was the de-facto library for Data Scientists. It evolves. Polars was a rewrite following bets practices learnt while working with Pandas.

I do not understand the aggressiveness towards previous evolutionary steps. Who knows, may they come up with newer methods. I still use it. I also use DuckDB and Polars. DuckDB is more useful to me these days from either. Still Pandas has a place in my arsenal. Mostly because I know it better and I like it.

There is also libraries for doing type checking.

> an Intel i5-1135G7 with 8 cores

The i5-1135G7 has only 4 cores. Hyperthreading is enabled, so it reports 8 "virtual processors".

Nitpick for the author: it looks like you've got (a,b] when you actually mean [a,b). Either that or change the ≥ to be >.

Sorry, where are you seeing this? In one of the code samples?

The tables in the https://eddie.codes/posts/pandas-should-go-extinct/#i-have-b... section.

If you mean the tables in the https://eddie.codes/posts/pandas-should-go-extinct/#i-have-b... section, it looks like those came straight from the Amazon paper, in which case __eddie__ shouldn't fix them, on general "quote accurately even if the quoted document has mistakes in it" principles. (Though a [sic] might help).

Title is currently, err...: Useful Code Comments

Hoping OP can fix this on their end so the url has the expected content. Whoops!

Yep, fixed now. Sorry!

I wonder what they taste like.

Pandas should go extinct? I’m confused Edit: Oh link was broken before

I would love to see a speed comparison with R incorporated into this.

With the exception of data.table (which has a painful API), anything else in R will be much slower. https://duckdblabs.github.io/db-benchmark/

You could still squeeze out some performance with cudf.pandas

I've plugged this several times here (just a fan) but chdb's DataStore is a "lazy" drop-in replacement for pandas dataframe. Pandas API with chdb performance. I don't use it myself (I do everything mostly in sql with either duckdb or chdb) but always found pandas more convinient than polars for quick and dirty data crunching (less typing and frankly more pythonic if you're used to slicing).

Agreed on all counts.

In many cases I’ve found directly using python primitives to be less confusing than pandas.

Similarly, in companies I’ve worked at, the datasets just aren’t that big. Especially if you’ve got access to modern hardware.

Data scientist here. I still use pandas. It is a work horse. But today, the progress in data science is much more in integrating sources and outputs. I find Grist (via API) and n8n much more applicable and process oriented than core DS pandas - although all of these components fit together somewhere. Most of the time, there is very little binding glue needed, e.g. a bash script or standard python file. DS is still fun, but it is much more process oriented today!

Flagged - clickbait title

import fireducks.pandas as pd

so this story is something that's really important for everybody to know about and should not get downvoted...

Wait, wait, wait... you're telling me that polars, something made in this decade, is better at modern problems than something made two decades ago?!

I'm shocked! Shocked, I say!

Sometimes you just want something that works with all the things. That's pandas. But, like the bamboo eaters, it wont be long...

Strongly disagree with the author.

For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well.

But if you need to construct dataset ? Or process existing dataset, do heavy filtering, transformation, reshaping, splitting? The proper ETL work, then pandas is really the perfect use case.

And pandas can work with small memory footprint as well, its actually trivial to do that, plus there are libraries like Modin that are upgrades over pandas with pandas api

Pandas is the swiss knife tool of data science that lets you do anything with the data and it integrates well with ML libraries

It's subjective, but IMO polars's APIs are better and than pandas's for ETL. Notably, pandas ported over polars's use of .col() for the 3.0.0 release: https://pandas.pydata.org/docs/whatsnew/v3.0.0.html#initial-...

Also, pandas's group_by() and window function pipelines made me want to tear my hair out while both are easy in polars.

Use pandas if you want advanced analytics, visualization, or ml.

Use SQL if you need to move data around.

How would you use sql to move data from three excel files (some of them may have arbitrary number of worksheets), pack of json files, two csv files, one mysql, one postgres db, some parquets, some of them in S3) ???

And in the end save processed data in aws s3 in another format like iceberg or whatever

In pandas a lot of these are one-liners that are impossible in sql (depends on what sql engine you have?)

Those are generally rules of thumbs for data pipelines. (Plus duckdb can read Excel these days I think).

Reply on news.ycombinator.com