It was daytime. One of our MySQL tables were growing at a rate higher that usual. Easiest way to check the size of the table? Duh, you `cd` into /var/lib/mysql on the database primary to `du -sh` the InnoDB table file...`SHOW TABLE STATUS` requires some arithmetic for true file size.

To this day I have no idea why I typed `rm -fr /var/lib/mysql` instead of `cd`. I blame muscle memory. Good news is MySQL pools its open files. This means the database was operational (but slowly failing opening less used tables) for 15 minutes or so.

We quickly promoted a database slave to primary and took the old one down for backup restore. I went for a teary walk. Shaken.

At the postmortem we concluded we needed more metrics from MySQL to avoid SSHing into servers. We also concluded how error prone one-off commands are. Personally, I'm much more careful nowadays.