Hacker News Reader

I really, really dislike all the dtype wrangling and how those choices resonate throughout the API. I understand that a lot of work has been done to make that API "work", but in practice it feels like that effort would have been better avoided by changing expectations and interfaces.

Now, to be clear, that's a hard problem. Heterogenous named bags of homogenous columns with a variety of data types, storage patterns, and ideas about missingness isn't an easy domain... but instead of just trying to make everything work through hammering 6+ semi-coherent interfaces (indices, databases, mutability, immutability/chaining, numpy, dataframes) together, I'd be willing to pay a lot more in verbosity and explicitness for something simple.

    pd.Series(str, [np.nan, "a"]) => ["nan", "a"] # or even an exception!
    pd.Series(nullable(str), [np.nan, "a"]) => [nan, "a"]

Indexing is vastly over-designed. GroupBy is a very common API and is poorly documented and just weird in no small part due to attempts at dtype inference. Foundational useful concepts like categories feel bolted on. There's join, merge, pivot, pivot_table.

I'd chalk this all up to just being "hard", but at the same time I can go pick up R's dplyr library and get a very nice existence proof of how a nice interface could work. Not to say dplyr has it all figured out, but it's a night-and-day improvement to Pandas.

Pandas is great. It makes doing data science in Python so vastly much less of a chore than working with straight Numpy. It steals some great ideas and tries out a few interesting ones of its own... but it is far from a joy to work with.

Agreed RE GroupBy being challenging, especially compared to dplyr.

As I've worked on a port of dplyr to python over the past year, though, I've realized the dtype issue (like you said), indexes, and GroupBy being difficult are likely connected. Basically,

  * dplyr can chop up a dataframe into 50,000 groups and apply arbitrary functions to it--no problem.
  * custom pandas grouped applies are very slow

There are basically three reasons for slow pandas apply methods...

  1. creating an index for each subgroup is slow (will not be a RangeIndex)
  2. initializing a series for each group is slow (mostly due to type inference being re-run; could be avoided)
  3. AFAIK more type inference is run when concatenating results

This leads to a world where grouped calculations can't be run using arbitrary expressions (e.g. lambdas), but have to go through specific SeriesGroupBy methods.

I wrote a bit on how I tried to work around that, to enable fast dplyr-like syntax over grouped data in python. Would definitely be interested in your take! There are other libraries, like ibis that do a good job with it, too!

https://siuba.readthedocs.io/en/latest/developer/pandas-grou...

I have been through this path of trying to figure out why pandas Groupby is slow. It is a big bottleneck when building packages on top of pandas.

Well, I'm a big fan of plotnine, and plydata was part of the inspiration for siuba!

I think at its core, the groupby issue is a really big problem, and am devoting most of this year to working on it. So if you ever want to pair to work on pandas / pandas wrapping libraries send me an email (link in profile)!

>GroupBy is a very common API and is poorly documented

My god is it poorly documented. A lot of us newbies trying to do dataframe stuff for the first time get stuck on GroupBy, trying to figure out how it works. Dplyr's functions are miles ahead on useability and documentation.

Hadley had the discipline to let go and start from scratch 2 if not 3 times before getting it perfect with `dplyr`.

plyr came before and I think there was something else

That’s how we got the amazing `dplyr`

I think pandas is well liked by those who move from C++ or Java, but is disliked by those who move from R

I agree - R has performant and robust dataframe functions. dplyr is great for small-medium sized datasets, data.table seems to be really performant for larger sets.

And now there is dtplyr, which simply creates a data.table back end with dplyr syntax on the front end.

I think this is key: there was a massive learning effort that went into the current dplyr interface. This risked fragmenting the community, but Hadley and his collaborators navigated those waters effectively. That's super tough work and R benefits significantly from it. If I had a magic wand for Pandas, I feel giving that team the opportunity to rework interfaces without killing momentum and community would go so far.

> hammering 6+ semi-coherent interfaces (indices, databases, mutability, immutability/chaining, numpy, dataframes) together,

That's a pretty succinct way of expressing one of panda's biggest pain points.

The API surface is huge because it heroically tries to handle all possible use cases. I do think it would've been significantly easier to develop, maintain and _use_ if pandas was more opinionated and offered a smaller set of very focused functions.

I've just spent the past 3 months reengineering what is essentially a spreadsheet calculator to use pandas. Pandas was chosen before I was there but it seemed like it was perfect for the job.

Aaand.. well, I guess it kind of is. But at the same time it isn't. Our biggest issues haven't been the complexity of what we've been implementing, but rather the insanity of the pandas interface. We've tried to keep it in mind that we're all inexperienced wrt. pandas, and it has definitely gotten better as we have gained experience, but that doesn't give us the hundreds of man-hours back we've spent trying to please pandas.

All the insane overloading of everything is awful and stupid. Give me 3 different ways to do 3 different things, not 9 ways to use 1 feature to do 1 thing.

I think another area pandas has done a lot of work on is with datetimes. Numpy's datetime objects are pretty deficient when you need to perform computations / data wrangling with them and utilizing python's native datetime objects would slow things down a decent bit. So they have done a lot of work to create their own datetime implementation that helps a lot when dealing with tabular data and performing date/time based arithmetic/manipulations.

Datetimes are still a mess. Now we have 3 datetime objects and converting between them is not obvious or trivial:

https://i.stack.imgur.com/uiXQd.png

Yeah sure, but that isn't their fault really. I can see a role for python's datetime being separate from numpy/pandas, but I do think a consolidated datetime object would be better to have rather than the numpy and pandas versions that are similar but not the same.

Definitely agree, pandas/numpy can have internal representations of efficient datetimes but the user should not have to deal with these conversions.

I use Julia and this is one area where Pandas still kicks the Julia ecosystem's proverbial ass: awesome support for working with nanosecond precision epoch timestamps.

I dare to promote one StackOverflow question [1] about pandas I have tried to investigate and answer [2] half a year ago. And I was rather horrified by its internal complexity after digging into pandas source :)

OP was wondering, why pandas facing a strange overhead after each 100th iteration in some very specific case. There was a proposal about Python's GC, but it was not clear at all.

Finally, I have dived into pandas and found that it has a hard-coded constant == 100 (!) of a number of internal data storage blocks. After reaching this value it runs some consolidation routines [3], and they consume a lot of memory even leading to crash with memory error.

What was much more wondering, is that after changing this constant to some large value (1000000, actually it disables consolidation at all) reduces memory consumption dramatically! This consolidation seems to reduce storage and memory consumption, so I still do not know why the opposite happens and why it works well in all other cases.

[1] https://stackoverflow.com/questions/56690909/python-is-facin... [2] https://stackoverflow.com/a/56705419/978424 [3] https://github.com/pandas-dev/pandas/blob/761bceb77d44aa63b7...

I like that pandas 1.0 took “Int64” out of trial mode.

It wasn’t functionally very different from everything being a float, but I will appreciate not having to format floats as ints in all my reports.

Ah, same! I'm also relieved to hopefully never again have to say: "whoops I converted nan into a very large integer".

Hi, I'm working on this if anyone is interested (work in progress):

https://github.com/dylan-profiler/visions

The key idea is to allow abstraction of physical types (boolean, integers) by defining custom semantic types (URLs, paths, probabilities). The idea originated while working on pandas-profiling [0] and running into similar problems. We found this abstraction to be effective for many other downstream tasks, too, including compression and AutoML. More coming soon...

[0]: https://github.com/pandas-profiling/pandas-profiling

There are always choices to make. In this case, I would much prefer to let the data be treated as is, i.e., no silent casting of np.nan to the string "nan". When dealing with numerical data, a string "nan" is rarely useful. But when you need it, you can still create a data series with a string "nan" using

    pd.Series([str(np.nan), "a"])

So you want the behavior that Pandas has (i.e., no silent casting of np.nan to the string "nan").

The point I wanted to make was that string "nan" is usually not as useful as floating point NaN when dealing with numerical data. Therefore, as default behavior it is acceptable, if one has to make a choice.