At some point one has to ask, what are you guys doing ingesting 5pb of data per day?!
Unless this is google, that doesn’t make any sense. That’s an average of 7.5mb per human on the planet, every day.
Unless this is google, that doesn’t make any sense. That’s an average of 7.5mb per human on the planet, every day.
Thinking about what to store (and what to log) is not trivial and takes careful consideration. Plus, there's always the argument: "But what if we need something that we forgot to store or log?".
Answering that takes time and risk-acceptance that most developers in most projects don't get.
"Datalake" was just a pseudo-solution to gain peace-of-mind: We throw everything into a big bucket and figure it out later.
Just like performance optimization, it's cheaper to buy more hardware than to pay for humans to think about it and coordinate.
Believe it or not this is just for security data. But that fact combined with SOA leads to lots of logs. (5Pbs is the uncompressed amount, but we decompress incoming data, then ETL).