We’re currently building real-time apis backed by terabytes of compressed parquet… hundreds of billions of ‘rows’… in exactly this fashion using polars. It amazes us at every turn.
Join us and help!
Join us and help!
Do you mean polars reading Parquet into DuckDB to process that amount of data?