Hacker News Reader

McKinney has been hard at work getting parquet and arrow support in pandas.

>Give the kind of people behind Arrow, I would love wrapper that will use Arrow to do all of this...But doesn't matter at the end of the day.

pyarrow; pyarrow.parquet (which uses parquet-cpp).

Wow this is great. I've been working around the jvm to integrate sklearn and some spark jobs that produce Parquet. This is a huge relief

Arrow doesn't do scikit - atleast last time I checked . Has it changed ?

pyarrow has methods to convert to pandas, which scikit supports

No - this is not it. Scikit models need to be persisted. The only ways I have found is pickle or dill.