Hacker News Reader

Comment by xtrumanx | Hacker News Reader

> We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one.

Wait, does that mean what I think it means? Did someone get an IIS splash page when visiting SO?

I was going to say they should live-blog during their next upgrade but it they did it on twitter which is awesome.

View on HN

Basically, yes. This happened: https://twitter.com/atimmer10/status/560568418945236995.

How long did it take to figure out it occurred? Do you have any stats on how many people hit that machine?

Would be hilarious if someone who went to SO to figure out why the IIS splash was showing instead of his app happened to hit that page.

It took about 20 minutes to notice and fix. The Twitter alerting system let us know really quick.

Why is their load balancer not configured to take servers like that out of rotation?

It normally would, except the base IIS page is a proper 200 to all requests. The load balancer removes all servers having issues - from HAProxy's point of view things were dandy.

As always, we are looking how to adjust our deployment process to prevent it from happening in the future.

Everyone I know big enough to run a load balancer has had this happen.

The next step in evolution is to have your script look for specific text on a page, which will change 24 months later and have all the perfectly good servers pulled out of rotation.

Does bare IIS respond with a 200 to /ping or /sping?

We generally pick a hashtag on twitter and do it that way. We've also done Hangouts On Air for non-hardware in the past which have been pretty fun as well.

If by fun you mean a 4 hour maintenance window turning into 8 hours ending well after midnight :-P

I never claimed to be sane

Reply on news.ycombinator.com