Hacker News Reader

Comment by StreamBright | Hacker News Reader

Or maybe an engineering mistake? I do not think you can claim anything without knowing the details. AWS lost customer data too, I would not try to assign blame without reading the post mortem.

View on HN

According to TFA (which apparently nobody reads) it most certainly was an engineering mistake. Files in long-term storage were accidentally subject to the same code that deletes files in short-term (30 day) storage.

It's sound similar to "junior dev had access to prod database" scenario which really isn't engineering mistake but process/administration.

I knew a guy who used a different color scheme for terminal sessions connected to production servers, so he didn’t do something in the wrong window.

A guy. One. How many people have I met with shell access to production servers?

I have all hosts share the same .zshrc and config with yadm, so all terminals look the same and share keyboard shortcuts. My oh-my-zsh with powerlevel10k is configured in a similar way: when I'm connected by ssh to a host it shows hostname@domain or hostname@ip on RIGHT side of a screen. It's not much, but it's easily visible, sometimes even annoying, but at least I never deleted wrong docker images from dev :)

It seems more likely that there was some process that performed a "delete older than 30 days" on the long term storage. Likely untested code. That's what the article says anyway, and we all know that the public message is always the best spin on a scenario.

An engineering mistake combined with an obvious lack of backups.

This reminds me of that oracle thread, where a guy had lots of backups, all encrypted with a key stored in RAM.

Reply on news.ycombinator.com