Why Pandas feels clunky when coming from R
sumsar.net
sumsar.net
In Python, I have been finding Polars nicer to use:
(purchases
.filter(pl.col("amount") <= pl.col("amount").median().over("country") * 10)
.group_by("country")
.agg(total = (pl.col("amount") - pl.col("discount")).sum())
)
SELECT country, SUM(amount - discount) AS total
FROM purchases
WHERE amount <= (
SELECT MEDIAN(amount) * 10
FROM purchases
WHERE country = purchases.country
)
GROUP BY country;
Using an appropriate DSL for the problem may be useful. In Python:
df.to_sql('purchases', db, index=False)
print(*db.execute(query))
# -> ('Canada', 270) ('USA', 8455)
Python as a glue language is one of its strong sides.
SELECT p.country, SUM(p.amount - p.discount) AS total
FROM purchases p
JOIN (
SELECT country, MEDIAN(amount) * 10 AS median_amount
FROM purchases
GROUP BY country
) m ON p.country = m.country
WHERE p.amount <= m.median_amount
GROUP BY p.country;I discovered that duckdb can make queries to pandas dataframes in SQL (postgresql parser) and do it faster than pandas itself.
Now I have zero motivation pandas queries.
Maybe for coloring or smth like that
So I get people love chaining and tidyverse, good for you, I don't. But at least I can acknowledge that my way (or this way) people have different preferences and one is not intrinsically easier.
Norm Matloff has a blog where he essentially just argues the opposite of all the tidyverse stuff, https://github.com/matloff/TidyverseSkeptic, but it is the same idea in reverse to me (one is not obviously easier to learn than the other IMO).
The biggest thing for me is that data frames are collections of series, and not collections of rows. That just seems wrong on such a fundamental level to me.
I commented a bit late on the earlier one so I’ll repeat myself here.
These comparisons feel like they are geared to analysts, not programming oriented data scientists. Which is fine, I suppose.
——- prior comment ——- This is fine and all, (although I’m not impressed by the quality of the python code), but the examples don’t show how to do meta programming.
I often dont want to manually write out column names, but programmatically specify them, and similar for a lot of other of these examples. I don’t want to manually configure them.
I haven’t seen examples of that higher level programming in these various R python comparisons. It’s always manual examples.
The examples usually feel like manual analyst query type tasks. The tone in this one strongly reinforces that with text like “oh and Maria asked me to xyz”
That is 80% of lowly data analyst work in a corp.
I use pandas regularly, but I’m not sure exactly what you mean here. I’m wondering if you’re referring to some techniques I’m not aware of that could be useful.
And there’s a ton of helpful things like dict unpacking of named aggs for groups bys. The dictionary can be the result of code or it can be spelled out.
Also, a side note, both pandas and R make it easy to chain or pipe a long series of ops. I try to keep the number down though. In other languages this can be frowned on as a hard to debug train wreck pattern, esp if it grows to long. Usually I define filters and functions outside the train and pass them in rather inline the filters or lambdas as they’re so much easier to debug that way.
.method2()
.method3()
) .method2()
.method3()
)