All 37signals Apps Were Down | Hacker News Reader

All 37signals Apps Were Down

status.37signals.com

71 pointsmef47 comments

I'm wondering how the red and green indicators look to someone who is color blind. Yes, the page looks nice, but I think Apple's status page is better simply because the indicator icons are (I'm assuming) better for people with color blindness.

Anyone who's color blind want to weigh in?

http://www.apple.com/support/systemstatus/

How common is color blindness? I'm not saying it shouldn't be a consideration, I've just heard it mentioned a lot and I'm curious if it's much more common than I had originally though (low single-digit percentages, higher among males)?

I'm red-green colorblind (deficient would probably be a better term).

What I've heard fairly often is the percentage ranges from 8-12% of males.

Seems like, in many cases, the number of colorblind people is higher than the numer of people using IE6. Would be a better use of resources to accommodate them than it would be to accommodate IE6 customers.

Of course, some colorblind people also use IE6, so there's overlap there, but is still rather code for colorblind people than IE6.

Colorblindness is something that can be designed around, but I'd be happy if I could find a plugin to alter a web page's palette.

EDIT: found this, but haven't tried yet: https://chrome.google.com/webstore/detail/chrome-daltonize/e...

It's ~7-10% amongst males depending on where you source your number.

Please forgive my total insensitivity..

3%+ of total population sounds like enough people to pressure at least one video card vendor to offer equivalent remapping. So why is this a web design problem?

That depends. Is your business doing so well you can afford to be too lazy to cater to 3-5% of the population?

(And what part of "design" doesn't include usability, anyway?)

Total color blindness (i.e. no functional cones, inability to distinguish any colors) is pretty rare, but some form of color vision deficiency is pretty common.

http://en.wikipedia.org/wiki/Color_blindness#Epidemiology

At my last company, the two other guys I worked with were both red/green colourblind. Luckily I'm not, so I could weigh in with opinions..

It seems quite common actually - I've met quite a few people that suffer it (not that this is anything other than anecdotal info, but...)

It will vary some between colorblind individuals. I can see the difference, both at apple and 37 signals. I do appreciate when people use a texture/hashmarks or shapes to differentiate as well.

I struggle immensely with the bitcoinity charts though. red, green, look nearly indistinguishable on the depth chart to me. And when I can tell a difference, I couldn't tell you if it was red or green. Just that it's a different shade than another bar.

The mnemonic I use for making color distinctions as clear as possible for as many people as possible is to just remember the orange/blue portal colors from Portal. Those work for most kinds of color-blindness.

If you were red/green colorblind, of the most common type (Protanopia), it would look similar to this — http://i.imgur.com/QoGzeBf.png

Photoshop has a proof tool which includes the two most common form of colorblindless. It is incredible how many interfaces become totally useless when you turn it on.

If you don't have Photoshop, there's a free app that let's you simulate your screen with 3 different forms of colorblindness.

http://colororacle.org/

I thought your second link was an example of a broken interface. You might want to change the line a bit.

They have a very impressive status dashboard. Clear, informative and dynamic. It refreshed itself while I was watching with progress updates.

It's also responsive! Looks great on mobile.

Agreed! It's one of the best status pages I've seen for a service (or set of services) like theirs. It's almost as if it's the AWS status pages run through a prettification machine.

Heroku's page is also great: https://status.heroku.com/

This is actually good PR for 37signals. We already knew that all apps go down from time to time. What we now know is that (a) 37signals has a great status page, (b) they are transparent about their uptime, and (c) that uptime is in the 99.98–98.99% range.

Looks like a load balancer issue: http://status.37signals.com/status/campfire/2013/4/17

I'm not too familiar with load balancers. I'm slowly learning rails server stuff that's more complicated than heroku.

Why would all of their applications be sitting behind a single load balancer? Wouldn't you want to silo them so that you do not have a single point of failure for all of your products?

All roads lead to a single URL (aka IP address) so somewhere along the way, you're going to have a single point of failure that does a one to many multiplexing, logically speaking.

Mason's question was about multiple URLs. IOW, "why does basecamp.37signals.com share a load balancer with campfire.37signals.com?"

It's usually for cost reasons. Hardware load balancers are expensive, and the job they do isn't especially computationally expensive, so it's generally accepted to host multiple VIPs on a single load balancer.

Looking at it the other way, being 100% down on one LB might be preferable to being 50% down when a single LB out of two fails. The alternative gives you twice as many chances for failure, even though each chance is half as "bad". Common-mode failure mechanisms would of course still bring you entirely down. (Bad config by a human being the most common cause.)

Most modern hardware LBs support active-active or active-passive connections with each other (deploying two A10s, etc.). It's been this way for 15 or so years (since local director, bigip, arrowpoint, etc.). The main reasons you would suffer from this would be misconfiguration or too cheap to buy 2 LBs.

(I do ops at 37signals...) We have two top of the line ("high end") load balancers at each site. They malfunctioned. When they both malfunction at once, it doesn't matter whether it's active/active or active/passive or anything like that.

> "why does basecamp.37signals.com share a load balancer with campfire.37signals.com?"

That is what I was trying to understand. Thanks for the insight.

That's not necessarily true. You can have more than one IP address serving the same hostname, and they'll get approximately round-robinned. The usual thing would be for these IP addresses to be on different machines. If you `dig www.google.com` you'll see an example.

But wouldn't you still get partial failure in that case?

Wouldn't that just increase latency, rather than induce failure, though?

I was waiting for someone who really knows the answer to jump in, but I don't think so, because a client would have to know to jump back to the DNS again after reaching the IP address failed, which I don't think they typically do. But I might be wrong.

Almost want to test it now...

Probably 80% of hackernews knows this by heart.

Well, if you have a single point of failure, even if it's a load balancer, you've done something wrong.

I will always refresh the page before posting a reply

Thanks Bart :)

You can have redundant load balancer setups that listen on the same IP [1]. Combined with round-robin DNS and there isn't a single point of failure unless your DNS servers go down.

[1] http://siag.nu/pen/vrrpd-linux.shtml

Not true: DNS servers can be configured with multiple A records for a single hostname, and the traffic will be spread round-robin. A DNS query will resolve to a different IP for different clients.

At Chartboost, our systems are configured with every DNS query hitting one of 6 load balancers, which in turn spreads traffic to ~200 app servers.

Sure but then the DNS server is the one to many multiplexer. You can shove things around all you want, you'll still end up with either a single point of failure or weak consistency.

Since DNS records change rarely, it's trivial to have multiple active DNS servers for a domain. In normal deployments, DNS doesn't have a single point of failure. It does have "weak consistency", like you say, but that doesn't matter if the records rarely change.

I'm a sysadmin at 37signals (was working during this outage).

The load balancers are meant to be redundant, and they are meant fail over automatically when something goes wrong. We're talking to the vendor about why this failed.

There are something like 89345793 different documents on how best to test Rails application code. Someone should write the canonical reference on testing the web infrastructure that people stick in front of Rails applications (really, all web applications).

I don't think the load balancers have anything to do with rails.

pretty sure tptacek is aware of this. ;)

I posted this code from here on Basecamp: http://stackoverflow.com/questions/6529728/html5-doctype-put...

Answer number one. It came in with the html with the stackoverflow profile pictures and all. I was like crap so I switched to non html formatting. I probably clicked a few times, it kept loading and loading. Basecamp went down right when that happened.

Could have been the problem...

It looks like it actually posts to basecamp servers to do the reformat. Why can't they just do this in javascript?

I am using Basecamp Classic with Firefox.

Painful yet interesting. I'm looking forward to the post-mortem on this.

Interesting that the "RSS" status link goes to rss.atom and is in fact an atom feed. RSS has more meaning I guess.

Google Apps was down this morning too, wonder if there are any similarities.

One of the best status page!

Reply on news.ycombinator.com