Good point. I updated the post with some more infos
What is your loss tolerance? If a file is gone, who is annoyed: a free user, a $50/year customer, or a $10k/year customer?
Are these files WORM?
Agreed - though I feel like every data use comes after the fact. Original software engineers/developpers rarely have the foresight that the data scientists need the information for (at least imho).
To be fair, the data scientists rarely have the foresight to know what the data scientists need the information for. The only time I've seen a data scientist correctly include all the data they needed (but still be wrong) was when they answered "All of it. We need all of the data".
So true. Tough to know in advance which data will hold the secrets.
And you can't build a time machine to go and get it once you do know. Want X days of historical data for training/backtesting and we just implemented the metric this sprint? Good luck meeting your deadline!