Hacker News Reader

Comment by yRetsyM | Hacker News Reader

I'm not very educated on this end of the spectrum - but I wonder if a process is possible where a rule or router update of some description is applied to one router only, testing the specific schema before pushing to the rest of the routers, thereby failing one router and not failing the rest? I understand the need to respond as quickly as possible - but as stated in this case this was already a manual response.

It appeals to my limited knowledge and non-existant experience that this would be a solution to the prevention of this occurring again in the future?

View on HN

Don't sell yourself short: our ops team has been on our internal chat talking about how to do something exactly like this for the last hour or so. It's difficult at our scale to truly simulate traffic, but we should be able to roll rules out to just subsets of our network. That's already how we handle router OS upgrades. If a small handful of data centers had crashed, likely no one would have noticed because we've designed that fault tolerance in. This was a problem because the crashes happened system-wide. In the end, we hadn't anticipated that a simple filtering rule like this would cause such a router crash, which was a bad assumption on our part.

Thanks for responding Matthew, and I can imagine the sense of frustration you're all feeling.

My 2cents:

Any change should be considered dangerous, and be tested first, time weighted to it's level of change. (An internal policy that could be communicated publicly. One that I apply to all my staff)

It would be also good to have data center clusters (preferably a datacenter clusters are sharded among regions) which would allow these changes to happen as necessary. A random cluster being the "first cluster" with a roll back in place if fails, or proceed through to other clusters progressively until all are live.

The sharding should hopefully alleviate any corner of the world taking any massive hit due to degraded performance.

Similar to yRetsyM, my domain knowledge doesn't extend to this side, but if your network is undergoing a DDOS or some other form of attack, taking the time to test rules in a pre-production/test server seems to be quite dangerous. What does the ops team think about using multiple hardware vendors?

Doing the wrong fix is at least as dangerous as not handling the DDoS (as shown in this case). Based on their general network architecture, I would think a prudent thing would be a quick sanity test on a pre-production system if available, then deploy to the various colos in groups at intervals that seem appropriate given the nature of the change. If pre-production isn't available, then having the first group be one colo limits the production impact.

If they did some colos as vendor J and some colos as vendor C, I think it would be manageable, but I don't really know how much of the cross colo traffic is actually their routers talking to their routers. Homogeneity in networks makes things easier to manage, until a platform fault breaks everything at the same time. In this case, at least it was related to a change they had made and happened quickly, so it was easy to determine the cause; other platform faults may not be as easy to determine, but if only your vendor X colos fell over, at least you'd have your vendor C colos up and something to look for.

Good idea. There might be a number of rules that can limit their communication with the routers (and even OOB)

Reply on news.ycombinator.com