Hacker News Reader

Comment by cageface | Hacker News Reader

This certainly won't be the first time that a system designed to increase uptime actually reduces it. I've seen a lot of "redundant" systems that are actually less reliable than simple standalones thanks to all the extra complexity of clustering.

I guess at Facebook's scale you have to build in fallbacks but this is a reminder that you can easily do more harm than good.

View on HN

Very true - the complexity of the system also comes into the equation when things go wrong and it takes real people to figure out why an outage is happening. Highly complex systems imply longer debugging time, and at a certain point a theoretically lower up time can give you higher practical up time just because engineers can actually understand and debug it.

Sometimes cluster systems are mistaken for high-availability solutions while they are actually load-balancing solutions and can decrease availability due to added complexity and dependencies.

Sometimes designers and builders of a system make decisions about tradeoffs between availability and scaling and error recovery that they don't understand until after the system is in the field. Any sufficiently complex system becomes vulnerable to unexpected interactions between subsystems especially when one or more subsystems fails catastrophically. http://en.wikipedia.org/wiki/System_accident has more on the topic, but you do want to read Perrow's book http://books.google.com/books?id=VC5hYoMw4N0C&printsec=f... Normal Accidents.

In am not sure that increasing uptime was a goal for this system. The way I read the story, this is about a system designed to make it easier to manage a server farm that was misconfigured and not robust against such configuration errors. So, it is more about a system designed to keep a server farm consistent. Staging those configuration changes could be a way to decrease the effects of future such errors, as it gives operators time to notice that something is amiss.

The other thing is that systems that cut across your architecture need to be handled with utter paranoia. They'll be the coupling points that cause global outages.

This is exactly why I've always been skeptical of extensive unit testing. Certainly you want some but the bugs almost always seem to crop up at the integration points. I always try to focus my tests on as much of the stack as I can reasonably address.

Of course, doing whole-stack testing in a meaningful way at Facebook's scale is a very tricky thing.

Unit testing does not preclude integration testing, they complement each other excellently.

It's not an either-or thing, I agree, but I see so much emphasis placed on unit testing lately, often at the expense of integration testing, particularly in the ruby world where people mock and stub so much of the code that you're testing something very far removed from the real runtime.

I don't think a lot of devs (perhaps especially the TDD faithful) appreciate how clumsy, immature, and downright primitive modern testing and mocking tools are. Even the best mocking frameworks give you nearly free reign to make up any behavior for mocked objects, many devs don't appreciate the importance of the gap between mocked behavior and actual behavior.

Which makes integration testing all the more critical.

Reply on news.ycombinator.com