Wordpress Hosted Blogs are Down
centernetworks.com
centernetworks.com
(Sadly, I did discover the outage when I opened my morning read of Fail Blog / Graph Jam / I can haz cheezeburger / Oddly Specific / Photobomb / WTF Pictures / Ugliest Tattoos / Failbooking. Does admitting that mean I'm no longer qualified for HN?)
This should have come under Don't Ask Don't Tell....
-Martin from Cheezburger
At a certain point (a point which TC passed quite a long time ago) you simply have to own your own infrastructure. That does not make it bullet proof, but it puts it under your own control and you can respond accordingly and better control the outcome of any outage.
I know that in Web2.0 everybody wants to think that they can outsource all the messy bits of running a website to the lowest bidder, but that approach has never worked.
When you get REALLY big (like facebook and google big) you move from a self-managed server in a carrier-neutral datacenter to building and maintaining your own datacenters.
For some reason people think it's perfectly normal that facebook builds its own datacenter, and yet also perfectly normal that a large blog (Techcrunch) take no ownership of the very equipment that makes their site go.
Once you own it all, you're still banking on the fact that you can do a better job than some company that specializes in what you're trying to do.
You'd be quite a bit better off sticking with some "cloud" provider and putting up a static failover cache on EC2 or something. Even then you're at the mercy of your DNS provider (unless you're going to run your own anycasted DNS) and any misbehaving DNS caches.
It makes sense for Facebook and Google to build and own their own data centers because they deal with volume on an hourly basis that is magnitudes larger than TC probably sees in a week (or possibly even a month). Techcrunch is a blog. That's it. That doesn't require owning your own datacenter. It requires a hosting company that can keep the lights on. Wordpress is about as good as you're going to get in that situation.
I don't think they should/would do that. They are nowhere near large enough in traffic volume to justify that. But they SHOULD have more direct control over the servers that run their blog so that WHEN something happens (again, I never claimed that maintaining your own server made you immune) they can directly control the outcome.
Chances are the downtimes don't cost them nearly enough to cover that.
No, you can; explicitly, legally, with a contract that states that any failures will be fiscally compensated for by the hosting company.
It's also worth considering how hard it is to cold-start a large site ...if your architecture is flawed.
http://www.nerdkits.com/videos/digital_calipers_dro/
So looking at our server logs now, we're largely seeing hits to that page from Google Reader (I guess it cached their RSS feed?).
Eventually I would think that they would belly up to the bar and implement a redundancy solution that involves multiple solutions. Then again, I am an infrastructure architect, not a blogger, what do I know?
Wordpress.com has a problem that is many magnitudes more difficult and complex than just keeping one blog available. Techcrunch could do a far better job by keeping their TTLs low, choosing a secondary datacenter site for replication. In the frequent event of an outage they just perform a master promotion, change the dns records in order to get back online quickly.
You don't have a redundant site if all of your eggs are at layeredtech. Throw SoftLayer or Equinix into the mix.