Hacker News Reader

Comment by kkwteh | Hacker News Reader

I was thinking of using Apache Drill for converting CSVs to Parquet. What else do people use for that?

I've also been looking for that. In an ideal world there would be a small, fast, standalone cli tool that can convert csv to parquet. There is a (sadly, unfinished) parquet writer Rust library in the Arrow repository that looks promising. All approaches I've tried so far (spark, pyarrow, drill, ...) require everything and the kitchen sink. So far I've settled on a java cli tool that uses jackson + org.apache.parquet internally, but it's cpu bound and has a huge amount of maven dependencies.

pandas + fastparquet fairly lightweight. but yes I would love to see a simple c++/golang binary that's just a simple csv2parq call.

Newer versions of Pandas don't even need fastparquet anymore. This code works:

import pandas as pd

df = pd.read_csv('data/us_presidents.csv')

df.to_parquet('tmp/us_presidents.parquet')

Nice! Does that work alongside reading in via chunks and writing via row_groups? If I have a 500GB CSV will it work?

This article explains how to convert CSVs to Parquet with Pandas, PySpark, and Koalas: https://mungingdata.com/python/writing-parquet-pandas-pyspar...

This article explains how to convert CSVs to Parquet with Go and Scala: https://mungingdata.com/go/csv-to-parquet/

If you're converting hundreds/thousands of files, Spark/PySpark is probably the best tool for the job. For fewer files, Python or Go is just fine.

https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

+1, in my experience the Pandas CSV parser seems very robust and more than sufficiently fast.

The only faster/better CSV parser I've used with any frequency is the fread function in the R package data.table. When I used R in the past, Parquet was much less popular, but I think now the arrow package supports writing to Parquet.

Just use a Apache NiFi fast and does that on 1 step

Spark

Spark community put a lot of effort into a feature-rich CSV data source. Often times the most challenging part of ingesting CSV is parsing it. There are many flavors each with different corner cases for NULLs, comments, headers, etc.

Maybe Presto/Athena

pyarrow

Reply on news.ycombinator.com