Hacker News Reader

Comment by stdbrouw | Hacker News Reader

Wes seems to be very focused on performance and big data applications these days, and of course it'd be great if Pandas could be used for bigger datasets, but when I hear people complain about Pandas they complain about:

1. the weird handling of types and null values (#4) 2. the verbosity of filtering like `dataframe[dataframe.column == x]` and transformations like `dataframe.col_a - dataframe.col_b`, compared to `dplyr` in R 3. warts on the indexing system (including MultiIndex, which is very powerful but confusing)

For those of us who use Pandas as an alternative to R, these usability shortcomings matter way more than memory efficiency.

View on HN

People interested in perf and big data ETL for the JVM may want to check out DataVec:

https://github.com/deeplearning4j/datavec

https://deeplearning4j.org/datavec

It vectorizes/tensorizes most major data types to put them in shape for machine learning. It also lets you save the data pipeline as a reusable object.

I too would welcome a friendlier pandas library, but every time I've tried to think of an API that would work I fail. Well actually, I keep on wanting pandas to understand SQL.

There’s a pandasql library which lets you execute SQL on dataframes. It’s a little slower because it needs to serialize via SQLite, but it’s a quick way to get going.

https://pypi.python.org/pypi/pandasql

I will definitely echo (2). dplyr is amazing and works in far, far fewer lines of code than pandas. That was my largest issue when migrating over from R.

There is dplython, but it doesn't quite work the same so I don't use it much. https://github.com/dodger487/dplython

I created plydata, you may find it sufficient for your needs.

https://github.com/has2k1/plydata

Is the only difference the placeholder X? I was running into issues with dplython in executing arbitrary functions outside the tidyverse. Can your package handle situations such as this?

df %>% select(var1, var2) %>% rbind(df2) %>% na.omit()

etc? That was the big benefit I saw from using dplyr.`

Yes, no X placeholder. And at the moment you cannot pipe to arbitrary functions, Python limitations. I'll get around this by providing a helper function e.g

df >> call(pd.dropna, axis=1)

#2 is a big issue for me. Filtering, subsetting, these are all really arcane-feeling transformations, there's a lot of weirdness with view vs copy, etc.

You can write 2) as dataframe.query("column == x").

Reply on news.ycombinator.com