Hacker News Reader

Have you considered something like iceberg tables?

Yes, but parquet hates small files.

You can't compact? i.e. iceberg maintenance

We might be doing something wrong, but we saw significant performance degradation for both ingestion and query when doing compaction when it comes to finance data during trading hours.

Reply on news.ycombinator.com