Hacker News Reader

Comment by walrus01 | Hacker News Reader

You'd be surprised - I work for an ISP that sells five nines SLA on most of its circuits, and meets it. We spend money to do it. Having 1+1 redundant core and agg router sets at every POP, geographic diversity of inter-metro fiber, full A and B side power/rectifier systems, etc.

With the right BGP and ospf design you can absolutely meet five nines availability for an end user customer perspective. We have some places that are approaching six nines.

This article reads like they had 1+0 everything and ran into a nasty iOS bug. Running out of ram is amateur hour as well.

When we need to do customer service impacting maintenance that will totally take their segment off the net, the hit can be from 15 seconds to a couple of minutes. And that is in a case where a colo customer is single homed to a single aggregation switch like one of our 48-port 10GbE aristas.

View on HN

Well aware of the pitfalls; I've worked in neteng for most of my career. But your statement of '15 seconds to a couple of minutes' - there's the rub. Do that more than once and you've blown your SLA.

Don't misunderstand me, I'm not defending Peakweb, simply saying that running a five nines network is hard - it typically requires deep experience in diverse problem domains.

If you read most any SLA - pre announced maintenance with advance notification is categorized differently than unexpected events, for contractual purposes. JunOS and IOS upgrades need to happen, crossconnects need to get moved, agg switches get replaced, etc. Customers know this.

It definitely requires ccie level knowledge and at least 10-15 years experience, plus advanced Linux/BSD server admin skills to really do five nines right. It is indeed expensive and requires enthusiastic cooperation from non technical management responsible for budgets.

I worked my way up from a level 1 NOC type position, so I'd like to think that after 20 years I have a good understanding of all the possible OSI layer 1 failure modes (and things you can fuck up in configuration at layers 2 and 3), yet the things some other partner and competitor regional ISPs do continue to surprise me.

It sounds like the two of you are in violent agreement. If it wasn't hard, people with your level of experience wouldn't be needed to get it right.

Nobody here are saying - as far as I can tell - that they shouldn't have done better. But that'd require them to actually have people with sufficient experience and the budgets and buy-in.

There's lots of hate for Peak Web here, which I totally understand, but it sounds to me like the problem really stems from Machine Zone shopping around for the cheapest possible contract. Maybe I'm just biased because I know that the game in question was basically just trying to sell desperate men discreet shots of Kate Upton's and Mariah Carey's cleavage.

I don't know the details of the countersuit, but that one exists at all is pretty telling. It suggests one of many things: it's possible that the contract did not guarantee any amount of uptime; or maybe Peak Web was not adequately informed of the load that the game was going to take on their servers and therefore they want to argue that the level of downtime was reasonable given that they weren't told to expect those kinds of loads; or maybe the contract had a termination procedure and Machine Zone decided to violate that procedure and just drop the company -- which is not something you can just do.

I mean, lawyers will argue anything for cash, but it sounds like Peak Web isn't exactly rolling in the cash they'd need to do this on a whim. I don't know what the chances are that the lawyers in question are working on contingency, but it seems plausible.

For a small to medium sized hosted to suddenly land a customer that needs $4m/mo of rack and power, or leased dedicated x86-64 servers, they almost certainly went into significant debt and/or leased a lot of the equipment themselves. I doubt peakweb has two nickels to rub together now.

Would you also encourage such a network be built with diversity of parts suppliers? I ask because it seems like you'd want to be able to also resist attackers exploiting a bug in a single vendor solution.

There are a great many ISPs successfully using a mix of juniper, Cisco, arista and other stuff. You will find extreme and foundry 1000baseT switches all over the place still. I would never encourage a monoculture of one model nexus 3000 or anything similar to that.

There's a lot of CCIE types out there who feel they have to maintain a totally Cisco shop regardless of equipment or diversity merits. This is finally beginning to fade, but I've seen it more than a few times.

Depends on the features that are needed. Cisco seems to have a quite a bit of cisco-only protocols that require Cisco gear. And for our particular use-cases, we seem to need those features. Then again, I'm at a ultra low latency shop that requires port-to-port switching at sub 300 nanos and faster.

If one is using standard protocols (bgp, ospf, etc) then mixing and matching doesn't really seem to be a problem.

Even OSPF/IS-IS interop is sketchy at best with some vendors, these days. Ah well.

Personally I'd never deploy a single model at the very least, when I am given the budgets for full redundancy, but ideally I'd prefer different suppliers too. I've lost too much sleep fixing problems where someone stupidly thought it was "simpler" to deal with the same everywhere. Until the same problem took down every router at once, or the same manufacturing defect caused their drives to start failing at a high rate at nearly the same time...

So it's not just attackers, but being susceptible to the same thing triggering the same bug by accident at the same time, or manufacturing defects across a whole batch or model, or affecting a part that's used all over the place.

Reply on news.ycombinator.com