Hacker News Reader

I wish it supported large csv to partitioned parquet. THAT is something I need a good solution for.

Are you familiar with fastparquet (https://github.com/dask/fastparquet) and pyarrow (https://pyarrow.readthedocs.io/en/latest) ?

Your link is malformed. https://github.com/dask/fastparquet

...and its fixed.

Not until you linked me. :) They look like good steps but not quite what I need, for different reasons. I'll have to check when I get back to the office next week.

(Py)Spark? I run it locally and do that kind of operation pretty frequently.

On what scale?

Part of the problem is the scale, but another part is that writing partitioned parquets seems poorly documented (I would love corrections, I spent a decent amount of time last week looking for good information)

I was recently working with a 38GB compressed text dataset locally. I can't remember how big it was in parquet.

It's literally one line of code. See http://labs.vistarmedia.com/2016/12/27/indexing-json-logs-wi... for an example (except you write to a local file system rather than HDFS).

That looks very helpful, I'll have to check it out!

Reply on news.ycombinator.com