Cisco’s Network Bugs Are Front and Center in Bankruptcy Fight | Hacker News Reader

Cisco’s Network Bugs Are Front and Center in Bankruptcy Fight

bloomberg.com

100 pointssytse39 comments

5-nines uptime means anticipating that the components (hardware and software) you bought from another company may fail in devious, correlated ways.

Failure to do so is simply incompetence.

Maybe the customer didn't read the fine print of their service guarantees, and that's on them. I would hope that doesn't happen often-- it would be very silly if service guarantees fell apart every time some piece of shoddy equipment (purchased and operated by the service provider) turns out to be at fault.

This.

The higher SLA system that I've created, was for a military project.

Physical network layout: I did choose a double port star topology, this is, every HP5300 modular swith, was connected to each other switch, with two "teamed" ports.

Usually if you only do the connections, you got a network loop. But with STP and VRRP in the HP 5300, I got an "always on" network.

The network did expand to +50 wifi access points with another proprietary wifi controller which also did have the capacity to gracefully failover connections from the APs, on network splits.

Servers behind the switches, where replicated in each segment. So you could at any time turn off (in order, or cutting the power switch suddenly) any rack (switch + servers) and the system did continue to work flawlessly. This was 5 identical racks/switches in star topology.

Acceptance tests did include to literally cut cables, literally turn off the UPS and RACK power, etc.

We could upgrade any firmware (the HP5300 cabinet, their hot swapable modules, or the servers BIOS/network-card/hard-disk firmware), without any service loss.

I'm happy with that result.

I've to say, all my other projects where I've work (+15 years), didn't have resources, neither did give any importance, to the needs of network firmware upgrades or downtime (because of a bomb?). In most cases, it was not because technical issues or handicaps, it was because of management.

Some projects did listen to me, and did contemplate the issue and planned it as a "maintenance window", or as what today is called "immutable infrastructure": prepare the new one, stop the service, replace, bring up the service.

I never did upgrade a switch/router firmware at $job, without having a backup switch ready and pre-configured, in case something went wrong. And preserve the backup one for a prudential time.

In my "always on" military project, firmware did need to pass acceptance tests in environments equal to production, before go to any production environment.

Edit: remove duplicated info

I used to do implementation of infrastructure on a project basis and found that every customer wants the best uptime possible until they get hit with the cost. Very quickly active/active failover proposals would turn to active/passive proposals with expectations of downtime if a failure occurred.

This web company promising such little downtime was stupid. And now they are bankrupt. Good.

I remember explaining to a storage customer why they needed three very expensive switches at the core of their network. One to handle the traffic, another to handle the traffic when the first was busy, and a third which was in the rack, patched and updated to one release behind the main switches, that could be swapped in by moving cables in under 5 minutes of operator time.

They initially thought I was kidding but then we did the fail tree on a white board. It was an interesting experience for them, understanding what it took to get what they took for granted.

You used the term "fail tree" which sounds interesting and I tried to learn more on the Internet but hit problems. Do you mean fault tree? https://en.wikipedia.org/wiki/Fault_tree_analysis If not, do you have a better search term and/or link I could follow up on?

Fault trees seem to be reasonably close. While developing my thinking and understanding on reliability analysis I came at it from an analog of decision trees. There were a couple of influential talks that got me started, one was by Sandia labs discussing the ways in which they insure that nuclear devices can't be detonated without approval (big requirement) through a process called vulnerability analysis, and another on how Citibank worked their network to insure uptime. The Chaos Monkey series from Netflix was quite fun as well. Google did something similar internally with its DiRT exercises (Diaster Recovery).

My "fail tree" (and I'll put it in quotes as unique to my conception of them) analysis consists of identifying a failure, the system response to the failure, a time to fix for the failed system, and a guess at the uncertainty on the fix. So for example "switch hardware failure" is a failure, with a fix time that varies based on "replacement part on hand" to "order/ship/install (replace) the entire switch". The first order is failure/fix tree with callouts of down time. The second order is mitigation/cost with mitigation strategies and their cost resulting in a new call out of potential downtime, and the third order is mitigation accelerators and their cost (which shorten recovery to non-degraded mode) which affect cost and possible down time.

Much of that you can do on paper, but sometimes you will have to run experiments to see how long things take to fix.

Thank you.

I'm surprised Peak Web isn't responsible for bugs in its suppliers stuff under the terms of the contract. After all, it picked them, and can switch to alternatives if they don't measure up, while the game company cannot.

If they agreed to an uptime of 99.995% (~27 min/year), then that's on Peak Web. It sounds like they oversold their capabilities. For that level of uptime, they should have had much more redundancy in their engineering.

The whole thing reads like peakweb are a bunch of 1+0 amateur hour "gee whiz we can be a hosting company too", and the customer wants a good reason to escape an onerous shitty contract.

I furiously agree with you. Peak Web failed their SLA in the first month, before the Cisco issue even cropped up:

> According to Machine Zone, the hosting service couldn’t make it a month without an outage lasting almost an hour. Another in August of that year was traced to faulty cables and cooling fans, according to the publisher.

"Three people familiar with Peak Web’s operations say the [10 hour] outage gave the company time to deduce that the troublesome command was reducing the switches’ available memory and causing them to crash."

Cisco Nexus devices log alerts for low memory/resource situations to syslog, the defaults being 85% minor, 90% severe, and 95% critical. Were they not reading their logs?

You should be able to build a lot of redunandcy into your network on $4 million a month, but that would eat into margins, wouldn't it?

Shit, I can build a lot of redundancy with four used/grey market $17,000/piece juniper mx960, $20,000 for rectifiers/battery... It's not THAT costly relative to $4m/mo even if your router requirements are as beastly as an asr9910 with 3rd gen line cards.

In networking, redundancy means two identical boxes with identical bugs.

There are very few companies that are willing to purchase network gear from two different vendors and run two completely different network stacks.

Management would be a huge pain in the ass... but it would be doable.

We are a large regional ASN in a tech heavy market and run 3 different vendors of DWDM platform and both Cisco and juniper in the IP core. Diversity is good.

Plus a third box for testing firmware updates before you touch the live ones.

Peakweb are idiots. With sufficient diversity and redundancy it is not that hard to run a five nines ISP. Ten hour outage is amateur hour. There are a great many terrible low budget hosting companies and colocation operators in the market.

Five nines is five minutes and sixteen seconds of downtime per year. That is an absurdly small amount of time, relative to an entire year. Saying it's not hard demonstrates your ignorance in operating a network.

You'd be surprised - I work for an ISP that sells five nines SLA on most of its circuits, and meets it. We spend money to do it. Having 1+1 redundant core and agg router sets at every POP, geographic diversity of inter-metro fiber, full A and B side power/rectifier systems, etc.

With the right BGP and ospf design you can absolutely meet five nines availability for an end user customer perspective. We have some places that are approaching six nines.

This article reads like they had 1+0 everything and ran into a nasty iOS bug. Running out of ram is amateur hour as well.

When we need to do customer service impacting maintenance that will totally take their segment off the net, the hit can be from 15 seconds to a couple of minutes. And that is in a case where a colo customer is single homed to a single aggregation switch like one of our 48-port 10GbE aristas.

Well aware of the pitfalls; I've worked in neteng for most of my career. But your statement of '15 seconds to a couple of minutes' - there's the rub. Do that more than once and you've blown your SLA.

Don't misunderstand me, I'm not defending Peakweb, simply saying that running a five nines network is hard - it typically requires deep experience in diverse problem domains.

If you read most any SLA - pre announced maintenance with advance notification is categorized differently than unexpected events, for contractual purposes. JunOS and IOS upgrades need to happen, crossconnects need to get moved, agg switches get replaced, etc. Customers know this.

It definitely requires ccie level knowledge and at least 10-15 years experience, plus advanced Linux/BSD server admin skills to really do five nines right. It is indeed expensive and requires enthusiastic cooperation from non technical management responsible for budgets.

I worked my way up from a level 1 NOC type position, so I'd like to think that after 20 years I have a good understanding of all the possible OSI layer 1 failure modes (and things you can fuck up in configuration at layers 2 and 3), yet the things some other partner and competitor regional ISPs do continue to surprise me.

It sounds like the two of you are in violent agreement. If it wasn't hard, people with your level of experience wouldn't be needed to get it right.

Nobody here are saying - as far as I can tell - that they shouldn't have done better. But that'd require them to actually have people with sufficient experience and the budgets and buy-in.

There's lots of hate for Peak Web here, which I totally understand, but it sounds to me like the problem really stems from Machine Zone shopping around for the cheapest possible contract. Maybe I'm just biased because I know that the game in question was basically just trying to sell desperate men discreet shots of Kate Upton's and Mariah Carey's cleavage.

I don't know the details of the countersuit, but that one exists at all is pretty telling. It suggests one of many things: it's possible that the contract did not guarantee any amount of uptime; or maybe Peak Web was not adequately informed of the load that the game was going to take on their servers and therefore they want to argue that the level of downtime was reasonable given that they weren't told to expect those kinds of loads; or maybe the contract had a termination procedure and Machine Zone decided to violate that procedure and just drop the company -- which is not something you can just do.

I mean, lawyers will argue anything for cash, but it sounds like Peak Web isn't exactly rolling in the cash they'd need to do this on a whim. I don't know what the chances are that the lawyers in question are working on contingency, but it seems plausible.

For a small to medium sized hosted to suddenly land a customer that needs $4m/mo of rack and power, or leased dedicated x86-64 servers, they almost certainly went into significant debt and/or leased a lot of the equipment themselves. I doubt peakweb has two nickels to rub together now.

Would you also encourage such a network be built with diversity of parts suppliers? I ask because it seems like you'd want to be able to also resist attackers exploiting a bug in a single vendor solution.

There are a great many ISPs successfully using a mix of juniper, Cisco, arista and other stuff. You will find extreme and foundry 1000baseT switches all over the place still. I would never encourage a monoculture of one model nexus 3000 or anything similar to that.

There's a lot of CCIE types out there who feel they have to maintain a totally Cisco shop regardless of equipment or diversity merits. This is finally beginning to fade, but I've seen it more than a few times.

Depends on the features that are needed. Cisco seems to have a quite a bit of cisco-only protocols that require Cisco gear. And for our particular use-cases, we seem to need those features. Then again, I'm at a ultra low latency shop that requires port-to-port switching at sub 300 nanos and faster.

If one is using standard protocols (bgp, ospf, etc) then mixing and matching doesn't really seem to be a problem.

Even OSPF/IS-IS interop is sketchy at best with some vendors, these days. Ah well.

Personally I'd never deploy a single model at the very least, when I am given the budgets for full redundancy, but ideally I'd prefer different suppliers too. I've lost too much sleep fixing problems where someone stupidly thought it was "simpler" to deal with the same everywhere. Until the same problem took down every router at once, or the same manufacturing defect caused their drives to start failing at a high rate at nearly the same time...

So it's not just attackers, but being susceptible to the same thing triggering the same bug by accident at the same time, or manufacturing defects across a whole batch or model, or affecting a part that's used all over the place.

Partial outages multiply the outage window by the fraction that it's available. So if you have an outage that affects 1% of traffic then five nines globally permits that 1% outage to last ~8 hours.

MachineZone shouldn't have put all of their eggs in a single basket. Multiple hosting providers, and write your server side so that failover can happen to other datacenters/locations almost instantly.

Exactly this. The idea that MachineZone was willing to take a business that they advertised during the Super Bowl (https://www.youtube.com/watch?v=XkaWyrm8EQg) and put it all in the hands of a single ISP at a single facility is questionable at best.

My experience has been that Cisco hardware from the high end to the low end is crap.

> The entire network often has to go down in order to patch—very disruptive in the best of times

This is why Erlang has hot loaded code, since around 1986. Those who xxx are doomed to repeat yyy....

Hot-loading code works in many languages. But it's not practical to hot-swap hardware driver code, especially in order to fix a bug that initializes the hardware wrong.

Also, the bug in question caused memory exhaustion, which isn't necessarily easy to fix by hot-swapping code without some kind of restart.

Apparently, it's five-nines week on Hacker News.

Reply on news.ycombinator.com