Network problems last Friday
github.com
github.com
For some reason, our switches were unable to learn a significant percentage of our MAC addresses and this aggregate traffic was enough to saturate all of the links between the access and aggregation switches, causing the poor performance we saw throughout the day.
Did you work with your vendor to understand what caused the above problem? Was it a lack of number of entries in the MAC table?
This problem aside, I am wondering why you still run layer 2 network in a tree-like configuration. These are known not to scale well, beyond a small LAN. An appropriate layer 3 network (with multipath routing) would ensure there is no such flooding, and ensure you use all the precious capacity in your switches!
"We have worked with our network vendor to provide diagnostic information which led them to discover the root cause for the MAC learning issues. We expect a final fix for this issue within the next week or so and will be deploying a software update to our switches at that time. In the mean time we are closely monitoring our aggregation to access layer capacity and have a workaround process if the problem comes up again."
In terms of network scalability, a properly designed layer 2 topology can scale quite a bit farther than our current needs. This is very much the least disruptive change we could introduce to solve the immediate network problems while we work on the future architecture.
Thanks a lot for sharing this bug story!
It's not really a tl;dr since (as far as I can see) it wasn't in the original post, though.
Well, I can say we experienced worse than that. Our private repositories were unavailable starting at 9am until 4pm PST.
Note: I initially forgot to mention that we had a single fileserver pair offline for a large part of the day affecting a small percentage of repositories. This was a side effect of the network problems and their impact on the high-availability clustering between the fileserver nodes. My apologies for missing this on the initial writeup.
Unfortunately, large L2 Ethernet networks is not scalable and prone to episodic catastrophic failure. You can read more here: http://blog.ioshints.info/2012/05/transparent-bridging-aka-l...
One way to make L2 network somewhat stable is to replace many little switches with one big modular chassis for hundreds ports, like Cat6500/BlackDiamond.
Or, to minimize L2 segments and connect between them in L3 (IP routing).
Given the flexibility to completely rearchitect our network you might see different decisions. Stay tuned. :)
Large scale networking changes like this are so challenging to pull off, one missed cable, one mis-configured switch, and blam! everything is nuts.
Only because industry best practices are incredibly fragile. A little DRY would go a long way towards eliminating misconfigurations.
It sounds like you rushed these switches into production, maybe with insufficient testing.
There are all kinds of bugs and weird interactions in network hardware and software that cause problems that you can't anticipate.
You have got to lab it up and do sanity checks before deploying (referring specifically to the lacp/port chan problems).
Lab testing is fine, and is a must, but it may or may not have caught this.
Datacenter agility is the only way to scale and have high availability.
True multi-datacenter availability in an active-active manner is actually fairly difficult to achieve.
"Where do we go from here" #2:
> With this in mind, we are planning to invest in a duplicate of our network stack from our routers all the way through our access layer switches to be used in a staging environment. This will allow us to more fully test these kinds of changes in the future, and hopefully detect bugs like the one that caused the problems on Friday.
You are ~100 million dollar company and it seems like you drew your systems architecture with crayons. The article is upsetting. The lack of segmentation is embarrassing.
"oh it's the switch's fault, it doesn't learn MACs fast enough" - actually you could subnet your racks and use f*n vlans. You might use public Ips on everything but this could still be educational for the company.
Your solution to all of this was to spend twice as much on a "staging" network. Something doesn't seem right here.
It makes me cringe when I see any one sentence that has the following three words in it: escalate, network, vendor.
This isn't a boeing airplane, you cannot just rely on the vendor. This article just gives me a good sense of job security in the field of sys engineering. I really think that they should sit down and really go over their network. A bridge loop like this for a company this large is pretty amateur. Github you can do so much better.
It is also nothing to do with the switch learning 'fast enough' and everything to do with the switch having a fault which entirely prevented it learning certain MAC addresses. Would you care to clarify how your solution (above) would help in this situation, and how you fix switch firmware issues without escalating to your vendor?
Hey github, sounds like you need SevOne. You could have diagnosed this issue with one TopN report and been done with it.
[/shameless plug]
edit: See following thread for a full explanation .
Or does the fact that I work at a company solving these problems preclude me from participating in the conversation ?
"Just do a TopN report" is not a diagnosis, analysis, or furtherance of the discussion. If you want to do an effective shameless plug, write a few sentences about how your product would have helped from the beginning of the incident, through its resolution. Show how it would have saved time/effort and given more visibility into the problem.
This was a multi-hour incident with code-level escalation to the network hardware vendor; assume complexity.
In this specific instance if you had an indicator on an OID .1.3.6.1.2.1.17.4.3.1.1 you could see how big the MAC table was and have noticed when it dropped below the expected value. This would have alerted immediately letting you know not only that you had a problem, but on which switches there were issues. Alternatively you could have set alerts on any number of packet level indicators.
If you had bothered to provide more actual information about what your software could have done for us, I would have reacted very differently.
This technically doesn't come out until Tuesday, but here's a peek: http://conferences.sigcomm.org/co-next/2012/eproceedings/con...
Just because those performing operations tasks are also performing development tasks doesn't mean anything substantial, other than it is more likely the Operations and Development teams are closer to being on the same page and there is less potential for someone getting to sit on their thumbs and spinning.
https://twitter.com/markimbriaco/status/276438853257162752