Hacker News Reader

Comment by keyist | Hacker News Reader

"a real serious business would have created an infrastructure that will provide paying customers a better redundancy for their accounts"

He should apply the same argument he makes against Github to his own business. Putting together a Gitosis or similar setup mirrored to Github would have given him and his paying customers the necessary redundancy to deploy when one of the hosts is down.

EDIT: Figure while I'm here and it's not OT I should give a well-deserved shout-out to Gitosis which has made my life a lot easier the past two years or so:

Gitosis setup is a one-off, very low maintenance install (you add repositories and users by committing config changes and public keys to a git repository -- hardly anything out of your regular workflow). Once set up, mirroring to github or another service is a one-liner in your post-receive hook.

View on HN

To borrow Jeff Bezos' analogy from Startup School, should all small businesses who pay a utility company for electricity also have enough generators to run their business should the utility company fail temporarily?

The raison d'etre for companies like Github is that small software companies can't afford to waste a lot of time building infrastructure. If they want to survive, they need to focus on their core competencies, their customers, and their products. If outsourcing infrastructure functions to Github doesn't allow you to do that, then there's not much reason to pay Github.

The electric company analogy is a good one, but I don't think you're looking at it correctly.

No service, not even electricity, has 100% uptime. If thinks are that mission critical, you need a backup plan. Should you disconnect from the grid and generate your own electricity? Probably not. But should you expect that it might go out at any time? Yes. If you are not running a UPS than no one is to blame except yourself. If things are so critical that a UPS won't last long enough, then you should have a generator backup.

In contrary, what would the poster have done if his power or internet had gone down when he needed to deploy? Would he be cancelling those services? For some reason I don't believe he would.

The real key to all of this is not a question of avoiding downtime, it's learning from it and avoiding a repeat of the same incident. I think GitHub has done a great job of this, and they're very clear about what happened and what they've done to avoid it in the future. The poster, however, fails at this point. Not only does he admit he doesn't have a backup plan for deploying when GitHub is down... he flat out states that he knows what he could have done, and REFUSES to do it. Instead he's just going to cut and run, thinking he'll find another service that somehow is somehow immune to unexpected downtime. I wish I could live in that world.

Moreover: Nothing comes without cost. If you want a service like GitHub but with higher reliability, you are going to pay. Possibly in money, possibly in ease of use, possibly in the time it takes to identify such a service (it's really hard to gather reliable uptime data, especially in a field that is constantly evolving).

And, all things being equal, I'm not anxious to pay more money for a more reliable Github. Because it's git, people. Why pay for five nines of uptime when you can just mirror your repo? Every five minutes, if you like? With a one-line cron task? One of git's most basic functions is to make an efficient mirror of itself.

Indeed, GitHub's terms even state "GitHub does not warrant that ... (ii) the service will be uninterrupted..."

If uptime was a serious factor for you, you'd probably pay for an SLA to ensure uptime and compensation for downtime... and a SLA wouldn't be cheap.

Also, just about every SLA is pure window dressing. I know this because IAAL.

One of our customers wanted an SLA with more teeth. They themselves with a straight face proposed an SLA that would give them ca 8 GBP in credit against hosting fees per day of downtime. This is a multi-million pound business. We said yes, of course.

We'd happily negotiate something with real teeth, as I'm sure most service providers would, and I'm sure Github would too if a big customer pushed for it.

But you're 100% right - if we did it'd be expensive, because we'd have to turn right around and insure ourselves against the risks incurred if the amounts were remotely serious, and we'd pass that insurance cost straight on.

Every Company I've ever worked for, in the last 13 years in the valley, has had both UPSes, and, when we got to north of 100 employees, a Generator.

The real question is, should every small company have _two_ generators. :-)

My employer is a local/regional media conglomerate. Newspapers, magazines, TV and a cable TV/Internet/phone service. So we have all sorts of fun toys to keep things running, including these:

http://www.flickr.com/photos/ubernostrum/255794358/

That's not a typo in the caption or the comments; each individual backup battery in that stack really weighs about 1500lb.

You don't have to generate electricity to use gitosis.

If I can't afford electricity to be down, the yes, the business should have generators. I would never put my hosted servers in a data centre without generators, for example, as the stuff I put there is there instead of in our office because it needs a solid environment.

On the other hand, our office servers are on a single circuit and we don't worry about it because a temporary electricity outage just means a few man hours lost productivity (rather than potentially large losses for our clients, leading to bad will and lost business for us)

If a Github outage is so critical to you that it's more than a few man hours lost productivity, then it means you're using it for something you either should have in house or should have adequate backup procedures for. And this holds regardless what you substitute for "Github" in this paragraph:

If it's critical, you ensure you have a backup, as it's virtually guaranteed that there will be some single point of failure outside of your (and theirs) control.

Although his example for cancelling github is a bit silly (Not deploying due to its downtime even though he could have set up a local repository).

His gripe with a paid service being down and unreliable and his action of cancelling is perfectly reasonable if he runs a business and wants a set uptime.

I think the point is that his alternatives don't guarantee 100% uptime either so the whole thing just comes off as some sort of tantrum.

  him: I needed to deploy but Github was down! Screw
       them! I'm going hosting company X instead.
   me: Does hosting company X guarantee that they will
       never be down when you want to deploy?
  him: ...

At least that's how it comes off to me. If those other hosting companies really do provide him with what he needs better than Github does, then more power to him, but I seriously doubt that they do at the price point they have set.

Even if they do "guarantee" it, what happens if they fail to meet their guarantee? Most likely they'll credit him with a refund of his hosting fees for the downtime, or if he's really lucky some small multiple of it...

Words are cheap, especially in the world of typical SLAs.

Gitosis is great, however I like Gitolite much better (ironically enough, hosted by GitHub - http://github.com/sitaramc/gitolite). Per branch permissions, and a much more intuitive project management.

Not really ironic because everybody hosts on GitHub now. The advantages far and away exceed the occasional downtime, unless you have very unusual uptime requirements as, apparently, the article author does.

I mean, I've been using GitHub 15 hours a day for a year, and last week was the first time downtime affected me, and even then I just couldn't push for awhile. That's kind of the whole point of decentralized version control right? So there isn't a single point of failure? I'd say the git ecosystem worked out just perfectly.

I'd say that if someone doesn't find GitHub's feature set to outweigh the level of downtime I've experienced, that person's use pattern probably doesn't fit the characteristics of a typical user. If I'm wrong, expect people to be leaving in droves now. My bet's on not.

I can imagine two situations where GitHub uptime is critical:

1. You're hosting websites via GitHub pages.

I'm hosting both my business and personal websites on GitHub. (Despite the recent outage, GitHub's uptime beats my previous el-cheapo host by a wide margin.)

2. You host client code and issues in a GitHub private repository.

This becomes an issue when clients want to add a bug to the issue tracker, etc.

I just use git init, ssh, and regular unix accounts, because it takes all of about 1 minute to set up, and is "the simplest thing that could possibly work". Presto, no more depending on external servers.

Agreed, given that you are dealing with Git, having GitHub go down shouldn't be a problem. Gitosis is nice in a simple way, but having a web interface to quickly/easily create repos, browse, and pass around url's is better.

I have been working on a little pet project called GitHaven (http://githaven.com) which is a simple web app that can be installed via a deb package. I have a test install running on http://git.meyerhome.net/ for those who are curious. I use to quickly push up private repos when I need to. Originally created because there seems to be a need for something between Gitosis and GitHub:fi that you can install wherever you need git hosting. If your interested in trying out GitHaven for your own development send me an email at ben@githaven.com

Reply on news.ycombinator.com