Why Pandas feels clunky when coming from R (2024)

That's my impression as well. Going back to the topic of the original post, pandas only partially implements the idioms of the tidyverse so you have to mix in a lot of different forms of syntax (with lambdas to boot) go get things done. Julia is much nicer, but I find myself using PythonCall more often than I'd like.

Scipy was originally supposed to provide the scientific computing stack, but then many offshoots in the direction of pandas / ibis / JAX, etc. happened. I guess that's what you get with a community-based language. MATLAB has its warts but MathWorks does manage to present a coherent stack on that end.

In fairness, if you're not touching Pandas, it's pretty good I'd say. Everything is based around numpy and scipy. Sklearn API is a bit idiosyncratic but works really nicely in practice and is extensible. JAX has an API which is 1:1 equivalent to numpy, probably with some catches but still. All the trouble starts with pandas.

Pandas is pretty terrible IMO for all the reasons listed by OP and TFA - and more.

There is so much hype and luck to widespread adoption, you never know with these things.

That's where I realised that the "modern" approach was taken in the article - which obviously I had not looked at.

1. Wrap the complicated bits in functions, then force it into the tidyverse model by abusing summarize and mutate.

2. Use data.table. It's very adaptable and handles arbitrary multiline expressions (returning a data.table if the last expression returns a list, otherwise returning the object as-is).

3. Use base R. It's not as bad as people make it out to be. You'll need to learn it to anyway, if you want to do anything beyond the basics.

So many different types of object, so many different syntaxes. The tidyverse makes sense, and sure, is elegant, but if your colleagues are using base R. Don't even get me started on docs and Stackoverflow for R. I much, and always will prefer Python.

The one area I still go back to R is on proper survey work. I've looked for years and haven't found anything equivalent to the survey package for Python. I do like that R tends to start from the assumption that data is weighted.

Fortunately I don't do surveys much anymore.

But, recently I’ve been working with much larger scale data than R can handle (thanks to R’s base int32 limitation) and have been needing to use Python instead.

Polars feels much more intuitive and similar to `dplyr` to me for table processing than Pandas does.

I often ask my LLM of choice to “translate this dplyr call to Polars” as I’ve been learning the Polars syntax.

    (
        purchases.loc[
            lambda x: x["amount"] < 10 * x.groupby("country")["amount"].transform("median")
        ]
        .assign(total=lambda df: df["amount"] - df["discount"])
        .groupby("country")["total"]
        .sum()
        .reset_index()  # to produce a DataFrame result
    )

purchases[amount <= median(amount)*10][, .(total = sum(amount - discount)), by = .(country)][order(country)]

```

- no quotes needed - no loc needed - only 1 groupby needed

   import pandas as pd
   purchases = pd.read_csv("https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-from-r/purchases.csv")
   (purchases
     .assign(country_median=lambda df:
         df.groupby("country")["amount"].transform("median")
     )
     .query("amount <= country_median * 10")
     .groupby("country")
     .assign(total=lambda df: (df["amount"] - df["discount"]).sum())
   )

https://pypi.org/project/narwhals/#description

I tried really hard to use Ibis but I ran into issues where it was way easier to do some stuff in pandas/polars and had to keep coming out of Ibis to make it work so I gave up on it for the time being.