Hacker News Reader

Comment by mrshoe | Hacker News Reader

These kinds of domino effects are one reason why scalability is so hard to get right. It reminds me of precipitation in supersaturated solutions. Everything seems normal until you reach some unforeseen tipping point, and then all hell breaks loose.

I like his little veiled pitch for Google's services when he talks about how easy it was to bring more request routers online given their elastic architecture. It makes me wonder why that elasticity isn't automated -- more routers should automatically be brought online if any routers hit their maximum load.

View on HN

Except that bringing new request routers online will make splashes too -- you'd have to throttle it, and even then you'd still have weird issues.

The correct solution, as outlined in their post, is for the servers to slow down when overloaded instead of trying to push load onto another server.

It's interesting to see this sort of thing happen again and again; I remember the first time I read the explanation behind AT&T's infamous nationwide crash in 1990, which was another domino-effect (a crashed switching station would reboot, broadcast that it was back, and the broadcast would crash other switching stations, which would reboot, broadcast that they were back...), and wondering how the hell you'd plan for or even test for that sort of problem.

I could imagine that this could mask bugs actually. Recently a project I work on started deadlocking. This meant randomly a single request thread would spin endlessly.

Now imagine we had a magic "start more servers" mode. We push new builds once a month, say. We'd just keep spinning up servers automatically without anyone noticing probably. Soon we'd hit some other system bottleneck (I'd bet something like too much DB traffic), and then we'd be off on a wild goose chase wondering what was wrong with the database or something.

(These are semi-hypothetical. As a team we discussed whether we wanted to have our monitoring processes spin up new frontends and decided that we should probably keep humans in the loop ;))

We saw the same thing recently, we became cpu bound before we expected to and as the load balancers compensated it increased the load on the others. All we could do was to start turning off expensive features until it evened out again. And add more capacity as fast as we could of course.

Reply on news.ycombinator.com