805 karma · joined August 21, 2012
[ my public key: https://keybase.io/jensrantil; my proof: https://keybase.io/jensrantil/sigs/lxsQ3_wfe5J1Akf75bO42UAhjVdv3TPY0WoT01wbMJA ]
[1] https://github.com/google/guava/wiki/ListenableFutureExplain...
To this day I have no idea why I typed `rm -fr /var/lib/mysql` instead of `cd`. I blame muscle memory. Good news is MySQL pools its open files. This means the database was operational (but slowly failing opening less used tables) for 15 minutes or so.
We quickly promoted a database slave to primary and took the old one down for backup restore. I went for a teary walk. Shaken.
At the postmortem we concluded we needed more metrics from MySQL to avoid SSHing into servers. We also concluded how error prone one-off commands are. Personally, I'm much more careful nowadays.