Siuba – A Dplyr Port to Python | Hacker News Reader

Siuba – A Dplyr Port to Python

github.com

77 pointsgrst30 comments

This looks neat, the _ trick is similar to the Self [0] of fastcore (from fastai).

However many things are possible with vanilla pandas. I use it a lot for data munging, usually with the fluent interface (method chaining) style [1], e.g.:

df.loc[lambda f: ...].groupby(...).agg(["mean", "count"])

It also plays nicely with the black autoformatter.

Anonymous functions are verbose and limited in Python, but you can still do many things and use a regular function when a lambda won't do.

I guess one of the thing I need the most when doing data analysis is column name autocompletion, inside groupby, lambdas, for column selection...

I wonder if one could do it in IPython, similar to the string autocompletion of file/directory paths. Basically a parsing of dataframe column names in order to autocomplete strings.

[0]: https://fastcore.fast.ai/basics.html#Self-(with-an-uppercase...

[1]: https://tomaugspurger.github.io/method-chaining

If the necessary information is there, as in you've mentioned a column name in a dataframe at least once, PyCharm will now do column name autocompletion. It's actually pretty solid in my experience.

Didn't know pycharm would do that... thanks for the info! Yeah the best things about dplyr in opinion are 1) less verbose than pandas 2) much better autocomplete.

IPython already does tab completion of data frame column names. E.G., `df[“col<tab>` will do what you’d hope.

As long as there isn't a space in the column name. This is riding on a Pandas trick of making the column name accessible as an attribute of the dataframe, which breaks down when there's a space.

Hey, thanks for pointing out Self--I definitely need to dig into fastcore more!

One motivation for developing siuba is that the grouped agg you show requires users specify only one operation on one column.

E.g.

1. Calculate mean of x

However, common operations like demeaning a column are multiple operations:

1. Calculate mean of x

2. Subtract result of (1) from x

In siuba you can just write mutate(res = _.x -_.x.mean()). This isn't possible from something like gdf.x.agg("mean"), and from what I can tell deeply confusing to analysts :/.

In vanilla pandas I really like to use the chaining method you laid out, and siuba to me is mostly a utility library for making the approach a little more succinct / performant[1].

siuba has experimental autocompletion (thanks to Tim Mastny!), and there's a pretty hefty technical write up on how it uses IPython machinery for that in siuba's architectural desicion record folder[2].

[1]: https://siuba.readthedocs.io/en/latest/developer/pandas-grou...

[2]: https://github.com/machow/siuba/blob/master/examples/archite...

Fastcore is great. I've been going ape with it recently for a work project, and I've particularly fallen in love with patch. Made it extremely easy to assemble a collection of pure functions into pseudo methods attached to an existing class.

What I think would be a nice approach would be someone creating a new Dataframe class that merely inherits from a Pandas dataframe but just implementing .filter(), .summarise(), .select() etc. as methods, using the dplyr syntax. If they return their own dataframes then chaining follows naturally.

I know that isn't how dplyr works, but it feels more Pythonic, and this solution isn't entirely analogous to dplyr anyway.

This approach seems a little complicated, though I'm sure with some use I could learn to enjoy it.

Hey, author of siuba here, I totally agree that subclassing would be a natural choice in python. One challenge there is that users will often get a DataFrame back (e.g. from pd.read_csv), so it requires a lot of casting to the child class.

Right now, a compromise I've been exploring is just attaching siuba's DF functions to a pandas DataFrame, e.g. df.siu_mutate(...). This seems to be what pandas wants people to do [1]! One obstacle here is that the DataFrame has 300+ methods, which can be overwhelming to learners.

I've spent a lot of time wondering whether the piping syntax feels like too much vs chaining. It's still an open question in my mind, so it's really helpful to hear what feels most natural!

https://pandas.pydata.org/pandas-docs/stable/development/ext...

What about monkey patching the class at runtime?

Edit: oh that's pretty much what the linked decorators do.

All good points! Great work on the project and thank you for working to make Pandas more friendly!

> What I think would be a nice approach would be someone creating a new Dataframe class that merely inherits from a Pandas dataframe but just implementing .filter(), .summarise(), .select() etc. as methods, using the dplyr syntax. If they return their own dataframes then chaining follows naturally. > I know that isn't how dplyr works, but it feels more Pythonic, and this solution isn't entirely analogous to dplyr anyway.

I think I agree. Pandas has a lot of overhead/baggage that I don't want 90% of the time. Being able to chain _simple_ verbs on a data frame would be great -- something like mtcars.groupby(cyl).summarize(avg_hp = hp.mean())

I'm still debating chaining vs piping, but you can do..

  from siuba import _
  from siuba.data import mtcars
  
  # mtcars is a pandas DataFrame

  mtcars \
    .groupby("cyl") \
    .siu_summarize(avg_hp=_.hp.mean())

Hey y'all, creator of siuba here--happy to answer any questions!

One piece of context I try to bring into discussions is that the way I test and develop siuba is by livecoding data analyses for an hour [1]. I encounter a lot of arguments like "X is possible with pandas", but when I sit down with analysts in realistic settings (e.g. time constrained) it turns out X works in more limited ways then they thought [2][3].

I'm a big fan of pandas though. It's what siuba is built on!

[1]: https://m.youtube.com/c/chowthedog

[2]: https://mchow.com/posts/2020-02-11-dplyr-in-python/

[3]: https://siuba.readthedocs.io/en/latest/developer/pandas-grou...

How's the performance? I'm certainly willing to give up a little computational performance for being able to write my code faster.

Using the experimental fast grouped pandas functions, it should run at the speed of optimized pandas code!

Since siuba functions just run on pandas DataFrames, you can always hand tune for performance, but imo most of the time pandas code runs slow it's because of something like .agg(lambda ...) somewhere.

There's an example with timings here:

https://siuba.readthedocs.io/en/latest/developer/pandas-grou...

Is the pipe operator >> specific to pandas objects or is it generally applicable? I'd like to see how you implemented it!

Dplyr and data.table are two libraries that make R shine in terms of data manipulation.

Python however has static analysis tools that are unavailable in R. I wonder if there is a data manipulation library in python that takes advantage of this. Looking at this library, it doesn't appear to use type hints. Other libraries, like pandas, have some basic support for typings but they are still far from being fully typed.

Don't forget ggplot2, another shiny tidyverse product. It has also been ported to python: https://plotnine.readthedocs.io/en/stable/

Siuba uses type hints to dispatch the appropriate versions of custom functions!

For example, siuba allows users to create custom functions using a thin wrapper around functools.singledispatch.

When deciding how to run...

    df >> filter(my_custom_func(_.x))

It requires that the return type be compatible with the backend being run (e.g. pandas, a SQL dialect).

Would be super interesting to try and lay out what would be needed to do static analysis via mypy. I think it'd require some plugins for singledispatch at least, probably some reworking things in ways myoy expects.

https://nbviewer.jupyter.org/github/machow/siuba/blob/master...

> A killer feature of siuba is that the same analysis code can be run on a local DataFrame, or a SQL source.

Everyone thinks this is a good idea until they discover that SQL is not actually very portable, and any attempt to make it so neutralizes whatever benefits your SQL DB has.

E.g. if you have a Postgres db, that has tons of excellent features that you actually want to use, sticking to a lowest-common-denominator flavor of SQL basically nukes all that.

I actually disagree, given that this is a dplyr port. dplyr has always had this feature, and I agree that it can make things difficult, but it can also provide a really, really useful abstraction over your SQL, especially if you need to change databases for whatever reason.

In general, this actually looks like the best dplyr port I've seen, and may actually get me to do some exploratory analysis in Python.

I think this is a port of the dplyr syntax/verbs only. None of the dplyr C++ code is used, it's all syntactic sugar on top of Pandas. So any SQL translation is going to be similar to what Pandas does.

siuba does the SQL translation :). pandas is used for local data, since it does a lot of optimization in c++, similar to dplyr's low level code.

Thanks for bringing this up--the docs could be clearer here

Yeah, I probably used the wrong word, I didn't think it used the dplyr code, just that it looked much more like dplyr

kudos for doing this. Whilst I haven't used Suiba, dplyr in R is a very compelling way to do data munging. If Siuba brings the same thing to python, it's a very welcome addition to the ecosystem.

I know pipes split opinion; I'm definitely in the 'pro' camp. Chaining ops in a dataflow pipeline fits my mental model well.

Neat. I've been working on my own "piped-Codd" style system I call the "data algebra" https://github.com/WinVector/data_algebra

I use method chaining as the composing notation.

This is, what, the fourth? Attempt to port dplyr to Python? Unclear why this one will be any more successful than the others

This one looks much, much better than the others, as a long-time user of dplyr who's now a refugee in the Python universe.

Hey, creator of siuba here. I think siuba's big advantage is that it can generate SQL code.

The architecture necessary to pull off executing either pandas or SQL also makes it very extensible (e.g. to spark or dask in the future :).

https://siuba.readthedocs.io/en/latest/key_features.html

Reply on news.ycombinator.com