How FreshBooks Use Immutable Infrastructure Concepts to Manage GitHub Enterprise

Given you had to "meet at least the following requirements: It should recover from known AWS failure scenarios It should never lose our data It should be trivial to bring back to a known working state"

How exactly are you going to "never lose our data"? There was a fleeting mention of a snapshot but that protects only to last snapshot. Otherwise it missed to address the subject of guarding the data and its state.

The whole blog post is about spinning replacement node if "nodes in AWS can (and will!) go away in a moments notice". That's a very limited recovery case and even unlikely to be required.

If your instance is EBS based you can just stop/start it and it'll come up on a new hardware. You can automate it as easily as configuring cloudwatch alarm to do it for you. That's not an issue at all.

What you should really focus on is the case when EBS volume is gone or AZ/region is gone or data got corrupted. Spinning up new instances programmatically can be done through trove of different tools and hardly addresses your objectives. Also wanted to ask, given the subject, what's immutable about GHE deployment?

P.S. HN formatting sucks. I tried to re-arrange it a bit.

At GitLab we're thinking about adding Atlas/Kubernetes deploy directly to GitLab to make this easier https://gitlab.com/gitlab-org/gitlab-ce/issues/3286

I'm really interested to hear what other people think about this.

Awkward time to see this on HN; my Freshbooks account appears to be down.