Pandas Illustrated: Visual Guide to Pandas | Hacker News Reader

Pandas Illustrated: Visual Guide to Pandas

betterprogramming.pub

263 pointsnemoniac41 comments

Open articleView on HN

https://web.archive.org/web/20230127194856/https://scribe.ci...

The proper (free) link is https://betterprogramming.pub/pandas-illustrated-the-definit...

Ok, changed from https://scribe.citizen4.eu/pandas-illustrated-the-definitive.... Thanks!

Can recommend taking a look at Polars. Kinda a successor to pandas.

https://www.pola.rs/

Interesting. Seems to also take quite a few leaves from PySpark's book.

Neat. I love that there's a rust implementation. Types make everything better

Thanks. This is so clean and also fast! I just can't get used to pandas so I do most data wrangling in R and then bring the dataframe over to Python for analysis. But looks like Polars will streamline my workflow

on polars vs pandas:

https://news.ycombinator.com/item?id=34452526

There is a new 2nd edition of the very popular Pandas book, Pandas for Everyone by Daniel Chen:

https://www.pearson.com/en-us/subject-catalog/p/pandas-for-e...

This seems to be getting the Hug of Death, but this looks like the content:

https://betterprogramming.pub/pandas-illustrated-the-definit...

LOL. Those were not the kind of pandas I was expecting.

One of my daughters is a panda bear fanatic and I thought this would be a resource I could share with her.

Same! Although at first glance it appears to be an excellent example of clear, well-illustrated documentation.

Brains are funny things. I saw the link for the first time a few hours ago, while I was on a lunch break, and immediately thought about the animal. I didn't click the link but did have a thought of "okay, that's kind of a strange topic". I went back to work, did some data analysis using pandas. Opened up HN again, and I saw the link and my brain didn't even register that this was a link I'd seen before -- it immediately assumed that it was about the python library. I didn't even think about it until I saw some comments about the animal.

To her, I can recommend the book "Big Panda and Little Dragon." Nice illustrations and simple wisdom.

So, given the title and how stressful the last couple of weeks have been, I was sadly disappointed when this wasn't about drawing cute black and white bears.

I mean, data analysis is useful and all, but not what the heart wanted at the moment.

asking DALL-E for some Python Pandas might relieve our disappointment

Hang in there! I hope you find a little relief from the stress eventually. Take care and don't forget to take breaks in possible situations.

I do almost all of my day job in pandas. I consider myself very good at it. My number one recommendation to new data scientists learning the ropes is to just not use NumPy almost at all. I’m not sure where people learn it but they do all of this complicated nonsense. Just map simple Python lambda funcs with pd.Series.map and that’s most of what you need. Memorize your pd.DataFrame methods.

If your code feels like it dealing with a matrix and not a table, it’s probably doing something funny.

You lose a lot of performance not using vectorised functions. Maybe not an issue if you're only dealing with small amounts of data.

Series.map is vectorized.

Pretty much everything you need in pandas is as performant as you ought to need for doing tabular data manipulation in Python. Except dataframe.apply

It is not.

    df = pd.DataFrame({"foo": np.random.randn(100000)})

pandas map:

    df["foo"].map(lambda x: x * 2)

18.1 ms ± 109 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

pandas apply:

    df["foo"].apply(lambda x: x * 2)

17.9 ms ± 46.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Vectorised function, using underlying numpy operations:

    df["foo"] * 2

267 µs ± 11.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Use numba and it is, including on GPU :)

If by "vectorized" you mean: "able to delegate the task of performing mathematical operations on the array's contents to optimized, compiled C code." then I do not think you are correct (unless perhaps you are supplying map with a dict or Series).

Series.map is not compiling your lambda's to C and running it. If there is a built-in method available it usually will be faster. Notable exception are pandas str methods which devolve into Python code but generally with more overhead than map/apply.

I mean not writing your own loops. A built in function is indeed better. But usually not what you need. And also, readability > speed when the execution time is trivial, which is it probably should be for pandas scale data.

Hey, I also use Pandas every day and I would definitely recommend keeping up with your Numpy skills sharp. A lot of Pandas is built on top of Numpy so there's one good reason, but another is that it would prevent you from footguns like thinking Series.map is vectorized.

Check out polars.

Vectorized, choice between lazy optimization and eager.

What is your day job?

Data science consulting for business things. Most datasets < 1M rows

Those are small datasets. Also numpy != just operations on a matrix, but as mentioned above, proper vectorization can have profound speed differences for larger datasets.

They are indeed small. But most real world problems operate on small data.

And if it’s bigger you probably ought to be using a different toolset than Python altogether.

In my experience it is exceedingly rare to find a situation where you have > 1M rows of data and need to do tabular data manipulation, and it’s not coming from a managed database sort of setup where the manipulation could be better done in the database.

I think it really depends on the scale of data. If you're dealing with anything less than a GB, it probably doesn't matter all that much, but once you're dealing with larger datasets there is a pretty massive difference with using vectorized operation. Some of the pandas dataframes methods map to underlying numpy ones, but I don't believe that is always the case

With the availability of things like pyspark the grey zone between pandas scale and pyspark scale is small and uncommon though. Especially for the awkward tabular data manipulation tasks where you actually need to be mapping custom functions and what not.

Pandas built ins can cover everything with good performance except dataframe.apply imo

That and using dicts as maps.

It'd be funny if this were a sequence of photos of people beside themselves with frustration while they sat in front of a computer.

These are not the Pandas I was looking for waves hand

I thought this was going to be about panda bears ;(

For those who were expecting pretty panda bears, like me:

> Pandas is an industry standard for analyzing data in Python.

Interesting info

@dang can you replace the link with the original? https://betterprogramming.pub/pandas-illustrated-the-definit...

Yet another pandas tutorial. Got chatgpt now, thx.

Good luck with plausible hallucinated interfaces in your statistically-generated responses.

this happens so much when i query chatgpt for pandas related things. It straight up makes up function parameters and sometimes functions

Reply on news.ycombinator.com