Hacker News Reader

I think this remains a usability problem for parquet: most of the tooling sucks. Starting to get better, but parquet has been out for so many years and there are still few tools that natively let someone interact with the format. For years, most of the utilities were exclusively JVM based.

My favorite way to explore parquet is to make a virtual link through DuckDB where you can seamlessly treat it like a SQL table. Even browse it in DBeaver.

View on HN

I mostly ignored Parquet until I realized DuckDB could handle it - now I'm much more likely to poke around with a Parquet file because I've got a solid toolkit available in both the CLI and as a Python library.

A couple of notes I've made about that:

Using DuckDB in Python to access Parquet data: https://til.simonwillison.net/duckdb/parquet

Use DuckDB to convert parquet to JSON and then open it in Datasette Lite: https://til.simonwillison.net/duckdb/parquet-to-json

you can also use pandas in Python to access parquet data, and then i.e. convert it to sqlite

Naive pandas is going to limit you to RAM sizes when parquet is deliberately meant for huge datasets.

Anyone reading who is currently using pandas and running into this issue-polars is generally faster, and now supports (still new though) out-of-core workloads.

Reply on news.ycombinator.com