Hacker News Reader

Comment by gumby | Hacker News Reader

I remember that outage. It was finally blamed (as described in this brief) on phone switch manufacturer DSC. IIRC this killed the company. Their SLA with their customers was something like three minutes of downtime per decade.

DSC was our customer at Cygnus. They were interesting as a customer (tough requirements but they paid a lot for them). For example if they reported a bug and got a fix they diffed the binaries and looked at every difference to be sure that the change was a result of the fix, and nothing else (no, they didn’t want upgrades).

View on HN

> Their SLA with their customers was something like three minutes of downtime per decade.

That is insane. I really feel like modern SLAs are only getting worse - so much so that most companies fudge them, and try their hardest to never declare any sort of outage.

What do you mean, Microsoft 358 is an excellent product that has uptime for exactly what they’ve got on the label!

The Bell System standard was an extremely high Erlang number, which of course they strove to provide at the lowest cost which meant extremely high utilization rate on hardware which in turn meant extreme uptime (compare to the then contemporary PTT QoS even in major economies like France).

This is also why the software itself was designed with so many internal defenses and what I would consider an “immune system”. I’ve never seen anything like it even on an aircraft control system. That is mentioned in passing in the brief article but is easily missed if you don’t know what it’s referring to.

Most of what is done on the Internet at, say, “layer 5 or above” isn’t at all important so there’s no need for this level of SLA, but the actual backbone carriers do still carry SLAs at around that level. With packet switching it’s easier for them to provide than it was in the days of the 4ESS and 5ESS.

> The Bell System standard was an extremely high Erlang number

I wasn't aware of erlang the unit (measuring telephone circuit load) and at first thought this had something to do with the language.

https://en.wikipedia.org/wiki/Erlang_(unit)

Well, they are related in that they are both telephony related: the language was an internal Ericsson language for building fast switches and such devices. I am always surprised (though I shouldn’t be) when I see it mentioned in a non-telecom context.

Someone must have felt very clever when they concluded Erlang would be dimensionless.

  Calls per second X seconds per call = no unit.

But that's wrong. Clearly the number of calls ongoing maps on to lines being used for a circuit-switched network, or datagrams in flight, or whatever. Far from dimensionless.

  Call_starts per second X call.seconds per call_start = calls as the unit

Much more plausible, IMHO.

Worst are SLAs where you have to prove the outage as a customer like wtf

Most modern SLAs are worthless. The penalty is meaningless and/or "up" is carefully defined in such a way that even a service failing 100% of requests is "up" because it's responding, or is defined such that a single customer can have a total outage but the service is up because it's servicing others.

Networking is the last bastion of SLAs that actually seem to matter.

Telco is a highly regulated business. If you go down, the FCC and the state regulator are all over you. If you break 911 service, heaven help you.

As a result, five or six nines availability is pretty standard. Six nines means you can lose no more than 31 seconds a year. As a result, traditional telco had all kinds of cool tech in it, eg the Nortel DMS had live code patching as early as 1990 iirc.

At one time I worked at a vendor of IP telco switches, aiming to replace the legacy Nortel and Lucent with smaller tech. We had to learn some very, very hard lessons about reliability, but we eventually got there.

Today, I see cloud hyperscalers claiming that can run telco workloads, but I remain pretty skeptical until they can prove that they can switch a call, mid stream, from one node to another, without losing the audio, while transcoding it from one codec to another. Im not saying that public cloud needs to make the same tech choices re resilience, but today's web tech absolutely will not cut it.

> Today, I see cloud hyperscalers claiming that can run telco workloads, but I remain pretty skeptical until they can prove that they can switch a call, mid stream, from one node to another, without losing the audio, while transcoding it from one codec to another. Im not saying that public cloud needs to make the same tech choices re resilience, but today's web tech absolutely will not cut it.

The thing is, traditional telco can't do it anymore either. CenturyLink (aka Lumen) has broken 911 at least twice since 2018 when I moved into their territory. I've got the county emergency dispatch center in my address book, cause chances are, they'll do it a couple more times.

> For example if they reported a bug and got a fix they diffed the binaries and looked at every difference to be sure that the change was a result of the fix, and nothing else (no, they didn’t want upgrades).

That sounds pretty diligent!

Sounds like a pain in the ass, it means custom branching for eternity.

It was a pain, but they paid a hefty premium for this level of service.

We required that all other customers upgrade at least once a year (or maybe 18 months? I don’t remember).

According to a tech that actually applied tbe updates (Long Lines group in FB), “The DSC incident occurred in June 1991, unrelated to this AT&T incident.”

Thanks. It was reported otherwise in the news back then, but TBH I'd trust someone likely to have been the person responsible, or close to that person.

(it might sound crazy these days to trust "someone on the Internet" but really: what's the incentive not to tell the truth about something like this?)

Reply on news.ycombinator.com