Peakweb are idiots. With sufficient diversity and redundancy it is not that hard to run a five nines ISP. Ten hour outage is amateur hour. There are a great many terrible low budget hosting companies and colocation operators in the market.
With the right BGP and ospf design you can absolutely meet five nines availability for an end user customer perspective. We have some places that are approaching six nines.
This article reads like they had 1+0 everything and ran into a nasty iOS bug. Running out of ram is amateur hour as well.
When we need to do customer service impacting maintenance that will totally take their segment off the net, the hit can be from 15 seconds to a couple of minutes. And that is in a case where a colo customer is single homed to a single aggregation switch like one of our 48-port 10GbE aristas.
Don't misunderstand me, I'm not defending Peakweb, simply saying that running a five nines network is hard - it typically requires deep experience in diverse problem domains.
It definitely requires ccie level knowledge and at least 10-15 years experience, plus advanced Linux/BSD server admin skills to really do five nines right. It is indeed expensive and requires enthusiastic cooperation from non technical management responsible for budgets.
I worked my way up from a level 1 NOC type position, so I'd like to think that after 20 years I have a good understanding of all the possible OSI layer 1 failure modes (and things you can fuck up in configuration at layers 2 and 3), yet the things some other partner and competitor regional ISPs do continue to surprise me.
Nobody here are saying - as far as I can tell - that they shouldn't have done better. But that'd require them to actually have people with sufficient experience and the budgets and buy-in.
I don't know the details of the countersuit, but that one exists at all is pretty telling. It suggests one of many things: it's possible that the contract did not guarantee any amount of uptime; or maybe Peak Web was not adequately informed of the load that the game was going to take on their servers and therefore they want to argue that the level of downtime was reasonable given that they weren't told to expect those kinds of loads; or maybe the contract had a termination procedure and Machine Zone decided to violate that procedure and just drop the company -- which is not something you can just do.
I mean, lawyers will argue anything for cash, but it sounds like Peak Web isn't exactly rolling in the cash they'd need to do this on a whim. I don't know what the chances are that the lawyers in question are working on contingency, but it seems plausible.
If one is using standard protocols (bgp, ospf, etc) then mixing and matching doesn't really seem to be a problem.
So it's not just attackers, but being susceptible to the same thing triggering the same bug by accident at the same time, or manufacturing defects across a whole batch or model, or affecting a part that's used all over the place.