Ask HN: Have you ever broken prod?

Not really my fault but I broke it once. I used to work for a 2-person startup where the only dev had almost no experience. I setup a database instead of putting all the data in the PHP code. Fast-forward 2 months, I'm still the only one with a testing database on my own computer, and I decide to delete my local data and start from scratch. The other developer starts complaining that the site is down, and I quickly understand that he plugged the "production" on my local database. Luckily I had a backup for this DB and quickly restored it. We only lost a few days of work.

The lesson is: make backups and make scripts to setup everything from scratch in a few seconds.

When testing our emergency backup restore process (Ansible, db-dump, fresh install and everything) on a dedicated server I missed the config line to define the server... Wiped the full prod server incl. service, db and everything but I was really lucky. The backup worked and everything was running a few minutes later.

accidentally turned off DHCP for the entire site at my first help desk job. thankfully i noticed before too many leases expired.