These are order and trade logs probably. You want to have them and you need them for auditing. Binance wants to be more professional in that way probably. HFT is making billions of orders per day per trader.
user ID: 128bits
timestamp: 96bits
ip address: 32bits
coin type: idk 32bits? how many fake internet money types can there be?
price: 32bits
quantity: 32bits
So total we have 352bits. Now let's double it for teh lulz so 704bits wtf not. You know what fuck it let's just round up to 1024bits. Each trade is 128bytes why not, that's a nice number.
That means 200Pb--2e17 bytes mind you--is enough to store 1.5625e16 trades. If all the traders are doing 1e9 trades/day, and we assume this dataset is 13mo of data, that means there are 38772 HFT traders all simultaneously making 11574 trades per second.. That seems like a lot..
In other words, that means Binance is processing 448.75 million orders per second.. Are they though?
EDIT: No, indeed some googling indicates they claim they can process something like 1.4 million TPS. But I'd hazard a guess the actual figure on average is less..
EDIT: err sorry, shoulda been 100Pb. Divide all those numbers by two. Still two orders of magnitude worth of absurd.
That's the only actual use case I can think of for something like this, which makes sense for a cryptocurrency exchange that is certainly expecting to get hacked at some point.
Medium retention (1 or 2 months) is still very appreciable if some issue in your bugtracker stay stale for this amount of time.
With a microservices architecture in particular that can pile up rapidly.
Also 100PB is measured as the input format (JSON). Internally Quickwit will have more efficient representations.