Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by hadley | Hacker News Reader

Full thread

hadley·

Both Wes and I (project authors) will be tracking this thread in case you have questions!

View on HN

Great idea! Some questions:

- Both R and Python support strings, factors, and complex objects in a dataframe. What is NOT supported by feather?

- Feather is "not for long term data storage". Will it be standardize in a distant future?

- Do you plan to integrate it into Pandas?

Feather currently doesn't support recursive/hierarchical data structures, like lists in R. That'll be added in the future though. We'll definitely standardise in the future so you can feel confident using it in the long term.

I have no plans to integrate it with pandas, but I'm sure Wes does ;)

Will you take a storage approach for nested data similar to Parquet?

Feather doesn't support a number of pandas features (but these are features that R doesn't really support, either): https://github.com/wesm/feather/blob/master/python/README.md (not comprehensive)

As one of many people flicking between R and python/pandas, do you feel there are other areas that have the potential for collaborative tools between the two communities?

I mentioned it elsewhere in the thread, but libdataframe.c seems like a pretty natural continuation to me.

EDIT: What I DON'T know is how much libdataframe would look than libsqlite.

Yes! Anything that is mostly C/C++ for performance could be shared between R and Python now that we have an easy interchange format.

In discussing Apache Arrow in the context of Python and R, we wanted to see if we could use the insights from feather to design a very fast file format for storing data frames that could be used by both languages. Thus, the Feather format was born.

Is the feather in insights from feather the right word? It reads awkwardly to me, which could just be me lacking context.

Should be insights from _arrow_

Reply on news.ycombinator.com