Bad script! Bad! So faulty, running itself with the wrong list of IDs and running itself in the wrong mode.
Bad script! Bad! So faulty, running itself with the wrong list of IDs and running itself in the wrong mode.
Lesson learned is not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong input/communication/decisions.
I doubt anyone can claim that for all such potentially dangerous scripts. But something that should be reconsidered for anything like a compliance delete script, even if the design intent is to specifically bypass the normal soft-delete process for something where it has been specifically decided that it should NOT be restoreable.
Fix in this case would be to make it a two-stage process: first an easily revertible soft-delete, evaluate the impact, and then a hard delete designed to only delete data that has already been soft-deleted. Never hard-delete active, live data directly.
It's also not clear from the post if someone had used the "permanently delete" flag accidentally before. It's very possible they had, but hadn't ever run it on a wrong id before.
While yes, someone ultimately did hit "enter" on a command with the wrong ids and the wrong mode, the existence of such a script enabled that mistake to have enormous consequences. Calling such a script "faulty" feels reasonable. In the same way having tooling which allows you to delete prod database disks without any additional safeguards would be called faulty.
In fact, recognizing that is precisely why you can’t even do that easily anymore. Setting aside that you probably need to sudo the function, rm prevents you from running the command unless you pass an additional long flag that basically say you really really want to do this.
Programs which expect perfect user input, and allow minor user mistakes to lead to major negative outcomes are faulty programs.
Nonetheless, even with rm you can override the check you mentioned with a flag. In principle, that seems to support the post you are referring to, not refute it (not in detail but in spirit).
It sounds like they have a "type system" problem, e.g. you shouldn't be able to give a script an ID for a "cloud site" when it was expecting an ID for an application. That is, this sort of outage should not have occurred because it should not have been representable to begin with.
When I do big deletions in an automated fashion, I always make it a two-step process.
The first step identifies exactly the items which will be deleted.
This gives you the user a chance to look at it and say, "Why are we deleting 100x as many entries as we expected?"
In many tools, this is as simple as running it twice, the first time with `--dry-run`.
In a microservices-centric architecture, it’s entirely possible that the script was used to orchestrate the change across a number of other systems, and an update on one of them led to to undefined behaviour and/or unforeseen consequences.