Hacker News Reader

Comment by mbell | Hacker News Reader

I noticed something like this several times in the article: "Each machine has 2x 1gbit ethernet ports, in this case we’re only using one."

It sounds like a single switch failure could take down a large amount of their infrastructure. The seeming lack of geographic (or even datacenter) redundancy also seems a little dangerous at this scale.

View on HN

There is detail about multi-site redundancy which we're saving for a future post :-)

You're right that a single switch failure would be impactful, but the impact of one switch going down would actually be very minimal. For example our web servers are spread out between many switches. The switches and network equipment have greater redundancy between them. - Avleen Vig, Staff Operations Engineer, Etsy

LAGs or redundant NICs is a mugs game. How deep do you go? Dual TORs up linked to dual aggs to dual transit routers, each with N+1 transit? all fed by A+B power infra, of course.

Or keep it simple. One nic for prod traffic, one for private/control plane. Separate TORs are nice, or just don't oversubscribe them. Keep critical data hot across racks. Plan for failure of any given host, then for dc failure. You'll have site power or transit failure waaay more often than a bad tor.

Yeah, that impresses me as a bit odd too.

I've seen situations where folks have done this because they believed they would need VSS/VPC or similar on the switch side, but ALB on the server should avoid that.

Perhaps they have a n array of access switches, each serving some subset to lessen the impact?

VSS/VPC == MLAG... but what is ALB? I am guessing server uses one link until failure then uses the other? Even then, there are still network issues to overcome that may justify MLAG.

Honestly, we find it's much easier to spread the load between many switches, and have enough capacity that one switch failure is a non-event. Switches are expensive, it's true. After a certain point it does become less expensive and complex to just add more servers and switches. There's a strong advantage to keeping things as simple as possible. Bonding isn't really complicated, but how many not-complicated things can you add before things become complicated? :-)

In fact while testing mrsync to synchronize search indexes among prod servers they overloaded their switches cpu and took down the site for a few minutes.

Reply on news.ycombinator.com