Hacker News Reader

Hacker News

Comment by datadrivenangel | Hacker News Reader

Full thread

datadrivenangel

Accidentally deleting prod is a rite of passage.

View on HN

Been there, done that.

The paradox is that every device I use interrupts me non-stop with modal dialogs ad nauseam.

But the production delete? You said,"drop", we said, "gone".

Either production database deletes need to adopt alert fatigue or the ops are already so fatigued that it doesn't matter.

I'm pretty annoyed by those prompts. Yes, fuck you, do what I fucking said, my backups work fine.

Also frankly I think most software does it wrong, if it is possible it shouldn't ask before delete but just allow to undo the step easily. Then work can go fast and uninterrupted while still being resilient to mistakes. Obviously not possible everywhere but where it is it should be the default.

Cool that you have a working backup, but restores are what counts. And they’re not instantaneous in almost all cases. So asking before certain destructive actions is still a good practice.

> Also frankly I think most software does it wrong, if it is possible it shouldn't ask before delete but just allow to undo the step easily.

Obviously. When undoing is not easy or possible automatically, there is the "Are you really really sure?" dialog.

>The paradox is that every device I use interrupts me non-stop with modal dialogs ad nauseam.

Pop-up warnings that are too common are a problem itself. People get so used to accepting them immediately that they stop serving their purpose and just become an annoyance. With one exception: they can prevent misclicks.

Right! That's why I used the term alert fatigue:

https://en.wikipedia.org/wiki/Alarm_fatigue

The misclick point is beyond truth, particularly given the storm of marketing that is mobile web.

In a company where we chose to do production SQL shenanigans in the spirit of "move fast, break things", I developed a habit of starting those with "BEGIN;" and paying careful attention to the number of records modified before calling it a "COMMIT;". In case of unexpected numbers, immediate "ROLLBACK;" so as to not hog the resources and make the applications suffer.

Even fatigued, "DELETE 1182900" tends to stick out, if you're used to, and expecting to, see "DELETE 1".

("BEGIN" appears to be PostgreSQLism, "START TRANSACTION" is more standard.)

Worked for a CRM SaaS. I once did a DELETE FROM users; WHERE...

Open office. Boss rolled over on his chair and started saying "so..." and I dismissed him with a "deleted all users leave me alone". He rolled away, kinda like Homer in that GIF where he disappears into the hedge. Luckily it was back up in minutes, but a new phobia was unlocked.

> and I dismissed him with a "deleted all users leave me alone"

I think I'm going to start unplugging my desk phone when one of those moments happen to me. "Hey, there's something wrong with..." "YES I KNOW I WAS FIXING IT BEFORE YOU INTERRUPTED ME"

I sort of did that at my first web-job. I thought it was a bit unhandy that the customers had to be mailed a link to the admin interface, why not just put a link to the login on the front page? A couple of days later I learned why. There's always a good reason somewhere, you just don't know why yet.

The reason? Well, the admin interface was written in ASP. According to the guy that called me in panic said that ASP version was so very old it was extremely insecure. I didn't have access to the database so could not verify anything but the webserver logs had a lot of "Bobby tables" in them so I have a strong suspicion it wasn't all ASP's fault...

Anyway, we ended up with the database disappearing in the night or containing random data until I got that phonecall asking why we were linking directly to the admin interface :-)

They were very good at restoring the database though, just took an hour or so every morning until they found the reason.

I dropped ~100 records from a join table thinking I was connected to the dev db.

"No worries, we have backups", I thought. Turns out those were more like replication, copying over every change immediately (including deletes).

In the end, fixed with a short script that cross-referenced tables and checked for missing relationships, but terrifying nonetheless.

Definitely! There's probably some mystical mechanism making sure newbies go through it.

I consider myself an "experienced" developer, yet on the first day on my current gig I dropped a '--recursive' flag in an AWS command and deleted a bunch of stuff I shouldn't have. ¯\_(ツ)_/¯

I’m sure some processes and procedures were updated on that day and the next.

At some time we learned about delete locks in Azure and promptly created those for all production resources. Have had/heard of enough actual/almost "killed production" stories to understand their value. A bit cumbersome when you actually need to delete something but so worth it for the mental health of the team.

The key with delete locks and similar is it HAS to be something out-of-band; if the normal way to delete is via a website, the delete lock removal should be API or command line, etc

A simple --max-really-delete added to a command line is subject to "replay attacks" on yourself when you use CTRL+R.

At a previous position a co-worker reached his rite of passage. Lots of connections, and not paying attention to which screen window/session he was in. After that, we then changed the bash prompts on all prod servers to be red.

Never had another incident..

Yeah, raises hand.

Even though I take all care to differenciate environments, always cross check, we had to live drop a table while on a meeting, and due to ongoing conversation the wrong window was selected, and multitasking ended up causing a major distraction.

Reply on news.ycombinator.com