Don’t scale: 99.999% uptime is for Wal-Mart
37signals.com
37signals.com
In my experience, you can get away with lower uptime but your users will crucify you for poor performance. Site down for a half hour here and there? Fine. Site responds slowly and your clients' data takes a while to update? Big, big problem.
The danger, of course, is that when something happens to cause a spike (news mention, post goes viral, etc), that's exactly when you can't go down.
For example, BCC has a substantial amount of functionality in the back-end interface so that I can handle common support tasks. AR has virtually nothing -- a single page which lists customer email addresses, trial statuses, and upcoming subscription renewal dates. I could have spent 2 weeks on building out a decent amount of functionality for CS and more advanced statistical navel gazing, but a) I might not pick the right stuff and b) it would mean that the release of the next feature that actually sells software would be on 3/15 instead of 3/1.
BCC has organically grown its backend over the years, as I get so frustrated with fixing the same issue manually that I make a one-button way to do it.
FairTutor will probably go live with no way to review teachers. Same reason.
You can't dump them if they're down!
So on the extreme end 37signals could be down for over 3 days in a single bout and still have 99% uptime. 3 days downtime is serious.
However, this case is unlikely.
Also unlikely is a scenario where they are down for the same time each day---14.4 minutes @ 99% uptime.
Its most likely that they're down for a few minutes here and there across the day during working hours, which makes for an incredibly frustrating experience. You can never just click a button and know your form will be submitted.
What people promise in an SLA and what they deliver are very different things. I'd be very interested in some stats about historic uptimes for similar sites.
Also, how much more are you willing to pay for an extra 9 of uptime?
The article contrasted Wal-Mart (massive e-commerce operation) with things like Delicious/Technorati (casual services, usually free to use). It conveniently glossed over the middle ground, where 37 Signals' applications and many others live.
FWIW, I did look into using some of 37 Signals' services for my small businesses. They failed my assessment on several counts and we found other solutions. But judging by their blog posts etc, they seem to be proud of several of the things that caused me to look elsewhere. As long as their marketing is good enough to maintain their business success regardless of any technical merit, I suppose I can't argue with their commercial position and I assume they accept that people like me will never use them.
The bottom line is, we didn't.
Most of these business admin services seem to offer little more than a very simple database, a pretty UI, and a monthly bill. We write software for a living ourselves, so producing an in-house bug tracker or CRM system or project dashboard takes a matter of hours, maybe a few days if we add more power to it later. It's only worth outsourcing the tools if the relatively small amount of time it saves us outweighs the many downsides: time lost to set up and maintain an external system, monthly cash cost to use it, having to fit our process around their tool instead of the other way around, interoperability issues if tools have related purposes but data is locked in, relying on external infrastructure for access, etc. etc.
That is only going to happen if the outsourced service offers a silky smooth set-up process, a tool that can be readily adjusted to our exact needs and still present a streamlined UI to match, immediate and convenient access to all the underlying data, and very low pricing, among other things. That's a pretty tall order, and frankly, none of the services we looked at seemed to be anywhere close.
We do use external software to run things like mail and web servers, but it's standard stuff running on our own boxes.
We are also looking at external services for tasks that are much more difficult/expensive to implement ourselves, such as payment processing and off-site back-up.
But for basic business admin, things like bug tracking, project planning, CRM and the like, we concluded that it wasn't even worth looking at the freebie software projects that do it. In the time it would take to properly evaluate a few of them and determine whether they could meet our needs, plus the time to customise them to do so, we could have just written exactly what we wanted from scratch.
I develop a web application that is used by schools and just can't entertain the notion of anything other than 100% uptime. I take the reliability of my product very very seriously. If one of my customers had a fire at their school and couldn't access our system for registers - that would be us and them up the proverbial creak without a paddle.
I've built up a company (over 7 years now) with a very good reputation for reliability and uptime. Don't assume that just because something is web based it doesn't require 100% uptime.
I'm not criticizing you or your product, but if reliability is critical, it's made explicit with a realistic SLA.
We don't pretend, promise or have an SLA that offers 100% uptime.
I couldn't say what the average uptime is, but if a school network goes down we normally get an immediate call asking us if there is a problem our end because they use our system so heavily. So at a guess .. at least 98-99% for the majority of secondary schools, but don't quote me!
Also teachers/students/parents access the system from home so we have to keep it running 24/7.
Computers have become mission critical in schools now - lessons, paperwork, planning, organisation, telephones, intercoms, security cameras†, locks‡ were all computer-dependent. To date that school network is the most advanced, highest-uptime network I've ever worked on: two server rooms (network continues at almost full capacity with the loss of either) in separate locations and a backup server in a third location. Unplanned outages were basically non-existent, and the loss of any service was very rare and involved lots of people shouting at us :(
† Internal only, external were still on tape
‡ I don't mean all the doors open if the network goes down, but the electronic locks were controlled over the network
Yeah, 100% uptime is a crazy notion; but something always to be strived for nonetheless.
An SLA only ever guarantees commercial remedies, not service availability.
Apart from one of the latest colo providers we use I've witnessed downtime with every single one that offered ip transit with a 100% SLA. That's why we use at a minimum 3 geographically distant providers. So far this approach has not let us down.
Reminds me of the "unlimited mobile phone internet" plans here in the UK which in the fine print resolve to "fair use policy of 1GB a month) .. why can't companies just call a spade a spade.
Slightly off-topic, but in case there's genuine curiosity there: AIUI, most of the advertising standards rulings about "fair use" policies related to Internet and particularly mobile bandwidth were made quite some time ago, when expectations were very different. It is possible that this was not entirely coincidental, and was motivated by the legal teams at major service providers who saw the current situation coming.
Can the teacher still teach if the systems go down? Yes.
I've had first hand experience dealing with dozens of LA (what you call districts) technical teams that take this kind of attitude and frankly I find it appalling.
I remember when schools where trying to bring in electronic registration, and the LA network teams just didn't give a damn about the schools needs. Instead they made there own conclusions as to what the school needed, and to them, having a reliable network so the school could move into the 21st century wasn't one of them.
Schools are becoming more and more engaged in technology, and downtime and poorly maintained systems set teachers and the work they do, back. Teachers need to be able to rely on the technology, simple as that, no excuses.
You're not expected to reach 100%, but to admit defeat and argue for that defeat is poor form.
We have had 3 incidents in the last 7+ years. The first 2 occurred when we only had servers in one site.
1, http://www.risktec.co.uk/news/business-continuity-%E2%80%93-... (lasted days!!) 2, A thunderstorm knocked out the power to our datacenter, the backup UPS and generators both failed to engage. (lasted a day, we also took our business elsewhere)
After the 2nd outage we decided to take up additional colo space elsewhere in the country. This has worked for years and any failures in our life datacenter have quickly been switched over to a hot backup datacenter.
We have had around a half dozen minor interruptions caused by server component failures ( psu / hard drive). This is because it takes a few minutes to detect the failure and switch over to a backup.
3, The last major failure was with our DNS hosting. They decided to scratch our main zone from there records because of a billing error. Luckily for us it happened on the Friday before a weeks school holiday. Needless to say we took our business elsewhere.
I'm sure we have had other instances of downtime I can't remember, but that's pretty much us in a nutshell. No idea what our uptime on paper would be, funnily.
I would also bet your uptime is worse than you think if you're not actively measuring it. However, I'm sure it is "good enough". I doubt it is worth spending an order of magnitude more money to get better than 3 9's for your business, which is more or less the point of the article imo.
I find the percent availability -> hours of downtime chart on wikipedia to be a really helpful reference for these types of discussions: http://en.wikipedia.org/wiki/High_availability
* 1 minute every hour, or
* 3h20 every week, or
* 1 week every year
I know about hyperbole for making a point, but does Basecamp really total anything like a week's downtime per year? If so, why? I'm pretty sure I've never had anything like that bad a number and equally that I wouldn't be happy using a service that did.
The general thrust is right: high reliability is expensive and you need to look at cost/benefit not chest-beating. But let's be honest about what we're actually aiming at.
[1] http://www.loudthinking.com/posts/31-myth-2-rails-is-expecte...
Its really easy to hit 98% uptime (2% downtime). Just take your eyes off of your homegrown solution for a day.
Sure, five and six 9's does get expensive, and that will depend on your cost of downtime (ie: lost sales, etc.).
http://www.amazon.com/exec/obidos/ASIN/0201699478/ref=ase_al...
Which is also why your average Web 2.0 application can't charge very much: without it, your comfort level slightly drops. No big deal.
While it may not be technically or fiscally trivial yet, it's far easier and cheaper than its ever been, and far more so that in 2005.
Somewhere in there is the pickle problem:
You have 1000kg of pickles in your basement. Now, pickles are mostly water. In fact, your pickles are 99% water, the rest is cellulose. Cellulose has negligible mass, so we can say that all the mass comes from the water. You leave your pickles in the basement for a year, and when you come back, they've dried out a certain amount, so they're now 98% water. What's their new mass?
"Now what if Delicious, Feedster, or Technorati goes down for 30 minutes?"
The article was written just six years ago, and these were the examples of popular sites that came to the author's mind. Gives you some idea on how transient this field is.
I agree with the post, but I don't like that it encourages settling for less than the best.
That's his decision obviously, but if I was a 37 Signals customer ever inconvenienced by problems with their infrastructure, I'd think of this article.