Hacker News Reader

Comment by domoritz | Hacker News Reader

If you need a quick tool to convert your CSV files, you can use csv2parquet from https://github.com/domoritz/arrow-tools.

View on HN

I think this remains a usability problem for parquet: most of the tooling sucks. Starting to get better, but parquet has been out for so many years and there are still few tools that natively let someone interact with the format. For years, most of the utilities were exclusively JVM based.

My favorite way to explore parquet is to make a virtual link through DuckDB where you can seamlessly treat it like a SQL table. Even browse it in DBeaver.

I mostly ignored Parquet until I realized DuckDB could handle it - now I'm much more likely to poke around with a Parquet file because I've got a solid toolkit available in both the CLI and as a Python library.

A couple of notes I've made about that:

Using DuckDB in Python to access Parquet data: https://til.simonwillison.net/duckdb/parquet

Use DuckDB to convert parquet to JSON and then open it in Datasette Lite: https://til.simonwillison.net/duckdb/parquet-to-json

you can also use pandas in Python to access parquet data, and then i.e. convert it to sqlite

Naive pandas is going to limit you to RAM sizes when parquet is deliberately meant for huge datasets.

Anyone reading who is currently using pandas and running into this issue-polars is generally faster, and now supports (still new though) out-of-core workloads.

We support exporting any query from Splitgraph to Parquet, try it here [0] (click the dice for a random query). You can also upload CSV or SQLite file(s) [1]. Each one will load into a table that you can query with a Postgres client, or export it (or just some subset of it you select with a query) as Parquet.

Then you can load it into Seafowl [2]. We also have an API to export from Splitgraph to Seafowl, just need to add the button (this week).

[0] https://www.splitgraph.com/query

[1] https://www.splitgraph.com/upload

[2] https://seafowl.io/docs/guides/uploading-csv-parquet

Reply on news.ycombinator.com