Hacker News Reader

Comment by dukeofharen | Hacker News Reader

Well, not recent, but a few years ago I deleted the production database of my fathers rental company (whoops). The (manual) backup was a week old. Luckily, all data was still there, but on paper (all signed contracts) so I worked overnight to restore all data from the paper pile.

After this, I built a script that did a backup once a day and made sure I had to go through some extra hoops to connect to the production database from my own dev machine.

All in all, this was a good learning experience.

View on HN

I hope you also periodically validate your backups with restoration as well - A backup is only good if it can be restored.

This! I've been backing up my Ubuntu machine using DejaDup (built in Ubuntu backups), only to realise - when my machine died - that they don't work. This was stupid on my part.

Talking of deletion...I mistakenly deleted production server, yesterday! I was trying to create some AWS micro instance in Mumbai region which were for some reasons were not accessible. Eventually I had to create a mini instance. So I was terminating all those micro instances and some where by mistake one of my important prod instances got selected and boom.

Though I had an AMI created few months back and it took around 15 mins to get latest code working but yeah I learnt my lesson.

But again I think this is a bad ux from AWS where you can directly terminate any instance without any prompt or warning. I hope they change it some day!

I'd suggest turning termination protection on across production instances as a first solution, and consider some sort of infrastructure as code in the long term.

I know how that feels. Even worse than any program or script, I especially dislike db management tools that just allow you to fire off random queries against the db. On the other hand you always get these little adrenaline rushes every time you do something like that until you can confirm that you actually fired that delete statement on the correct table... or not.

The trick is to execute a BEGIN TRANSACTION, then do whatever you need to, then verify that you've done everything correctly (consider doing this twice). Then, only when you're absolutely certain all is as it should be, do you execute a COMMIT TRANSACTION.

All of this is predicated on the idea that you know exactly what changes you need to make, exactly what state the database should be in afterwards, exactly how to verify this, and also how to check for unintended side-effects.

I suppose the real trick is to avoid having to do this kind of thing in the first place but, as well all know, sometimes that's just not possible.

Oh dear, I know that feeling. It's just a few rows and you've done it a hundred times before. You type in the delete query in a rush and look at it for a minute.. Just to make sure... Well OK: You hit F5 and then it runs... and runs... and runs...

(12'503'450 rows affected)

Oh crap.

That was an example of what could be called a teachable moment. I'm remembering my standout moment as I type, the 'oh no, oh no' as I realised what was happening.

You can avoid accessing the production database entirely, unless you need to alter the structure, by creating a local one - which you've created via the restoration of your last backup.

Reply on news.ycombinator.com