HNHacker News
TopNewBestAskShowJobs

big_whack

50 karma · joined November 1, 2023

submissionscomments
big_whack··on Understanding Parquet, Iceberg and Data Lakehouses
In my experience,

* Storing large amounts like petabytes in any database is phenomenally expensive, just for the storage alone.

* For some kinds of data, like image data, databases are generally the wrong tool.

* The consumers of these kinds of systems may have really dynamic workloads. Imagine ML jobs that kick off 1K machines simultaneously to hammer your DB and read from it as fast as possible. Cloud-managed object stores have solved this scaling issue already. If you can get infrastructure you manage out of the way, you get to leverage that work. If your DB is in the middle, you're on call for it.

big_whack··on Database Fundamentals
This is a good paper for a similar kind of overview: https://dsf.berkeley.edu/papers/fntdb07-architecture.pdf
← PreviousPage 2 of 2