> On or about June 9, 2022, Musk’s counsel indicated that granting access to 30 days’ worth of historical firehose data would satisfy Musk’s request for the firehose data. So, on June 15, the company gave Musk’s team secure access to that raw data — about 49 tebibytes’ worth.
From page 39. Holy heck. 49 TB of data? How do you even ingest that much data in a reasonable amount of time into a database.
Stripe 7 x 8TB drives into a RAID0 (should be able to get ~1.3GB/s write), slap JFS or XFS on top (for max file size), dump it all into an unindexed SQLite database (up to 256tebis), Bob is your database's uncle.