O2 'to seek millions' in damages over data outage | Hacker News Reader

O2 'to seek millions' in damages over data outage

bbc.co.uk

70 pointsbauc46 comments

Ericsson said last week that "an initial root cause analysis" had indicated that the "main issue was an expired certificate in the software versions installed with these customers".

I suspect I'll never know, but I would be really interested to know what sort of certificate. Was this part of some larger public system or was it a purely internal PKI? If internal, did the certificates _do_ anything of value or did they purely make the system more fragile because somebody thought it ought to have certificates?

One of the arguments in favour of abusing the Web PKI to do other things is that tooling for the Web PKI is in a relatively good place. And a lot of other things that you might ideally hope exist actually DO exist for the Web PKI. There's independent oversight, somebody is actually reading those yawn-making audit reports, it isn't perfect but it's not just make believe either.

But on the other hand, the Web PKI is ours, and so if it suits us to change it we don't really care that this is annoying for your payment card systems, jet aeroplanes, nuclear submarines or whatever else you've duct-taped the Web PKI into. This was fun to watch with SHA-1 for example and is currently causing some fallout for names with underscores in (DNS can handle a name with an underscore in it but they're prohibited for hostnames. Lots of people ignored that, but that's their problem, not ours and they are not happy about that).

Ericsson have both their own PKI infrastructure[0], at least for software integrity checking (however that certificate is valid since March this year, and the CRL[1] it refers to is empty), in addition to using other certificate authorities for everything such as the hosting of websites to internal infrastructure[2]. I suspect it wasn't any of the above - rather it is the PKI that is used in running the IPSec networking done between carrier's RANs and other parts of core network[3], which is probably Ericsson's own internal CA.

[0] https://www.ericsson.com/en/about-us/enterprise-security/pki

[1] http://crl.ericsson.net/Ericsson_Software_Deliverable_Integr...

[2] https://crt.sh/?q=%.ericsson.net

[3] https://en.wikipedia.org/wiki/System_Architecture_Evolution

Further expanding. GPRS Tunneling Protocol (GTP)[1] is what gets used to connected to the provider's data/voice network. This could be over any medium (wifi, GSM, UMTS, or LTE). It's likely this was the cert protecting GTP-C's ipsec tunnel[2] as without the ability to signal, pretty much everything on the network goes down.

[1] https://en.wikipedia.org/wiki/GPRS_Tunnelling_Protocol

[2] https://cyber-defense.sans.org/resources/papers/gsec/securin...

As far as I know GTP is only used for data sessions (PDP) contexts. It should not affect handset registration and circuit-switched voice.

Can you explain/link more about sha1 and underscores?

Can't seem to find anything on web.

Both underscores (because against standard) and SHA-1 (because old and weak) were removed from browsers.

On underscores: https://www.digicert.com/blog/digicert-pushes-underscore-ext...

On SHA-1: https://blog.mozilla.org/security/2015/10/20/continuing-to-p...

Mmm, rather than the behaviour being "removed from browsers" the important thing is that Certificate Authorities were told that issuing such certificates could cause them to be distrusted as a whole by browsers.

Part of the reason to do this is that if CAs issue the certificates then their customers buy them, and create pressure to accept them. If none of the CAs are willing to issue this never comes up.

Another is "unknown unknowns". Security systems don't play well with undefined behaviour, if we explicitly forbid everything we don't want to exist that minimises the risk of such undefined behaviour anywhere in the certificate handling code getting exploited. It's a defence in depth.

Wonder whose responsibility it was to update that, O2 or Ericsson.

Ericsson generally offers managed services, and so it's likely it was their responsibility. Also, this is reflected by O2 seeking damages.

It apparently only effected a new software release - so its down to Ericson in this case.

This is supported by the fact that Softbank in Japan experiened a similar network outage at the same time. They too blame Ericsson.

Wow!

At the same time Softbank's network was also down at the same time in Japan for the same Ericsson issue. The scale is quite surprising.

Softbank's press release [0] mentions that a total of 11 countries were affected by the outage. I can't seem to find any information on what other countries were affected though.

[0] https://www.softbank.jp/en/corp/group/sbm/news/press/2018/20...

Vietnam's second largest mobile network operator Mobifone was affected for 3 hours[1]. Now we know 3: UK, Japan and Vietnam. I'm sure it's possible track down the other 8.

[1] (link in Vietnamese) https://congnghe.tuoitre.vn/ericsson-xin-loi-ve-su-co-sap-nh...

Wow, Hong Kong's Smartone uses Ericsson and wasn't effected. I am surprised at how wide spread the problem is. How could Ericsson messed this up when the it has the biggest chance to grab business from Huawei.

This seems like a publicity stunt more than anything else. There's no way they actually lost 100MM here, and doing this ensures the public sees Ericsson as the real villain, and paints O2 as the victim.

It's a very clever strategy, nonetheless.

Just some dirty back-of-the-napkin numbers, but theoretically, £100MM doesn't seem unreasonable.

O2 have 25MM direct customers, plus another 7MM through reseller networks — so that's 32MM people directly affected by the outage.

Presuming O2 receives an average of £30 per customer per month (which I think is a reasonable estimate), that's £940MM/month, working out to £32MM per day in a 30-day month.

O2 are refunding customers 2 days' worth of service for the outage, so that would be £64MM in lost revenue.

Add on top of that brand and reputation damage — and other factors I've no doubt forgotten to include — and £100MM doesn't seem unreasonable.

I think you forgot that O2 won't lose that money because they never refund to their customers. But I might be wrong. So basically the clients lose the money.

Also average monthly can't be so high maybe you're including the device cost as well.

Remember that many might have prepaid and no contracts.

I'm with O2 and was affected, and they have messaged me promising credit of 2 days of my airtime bill.

What does MM mean? Google wasn't able to help.

Millions plural

Just to clarify - MM actually just represents 'million' where a single M represents 'thousand'.

The plural part of your definition here is not really relevant.. i.e. 10M = ten thousand, 10MM = ten million, 1M = 1 thousand (not 1 million, singular)

Most people colloquially seem to use k for 'thousand' and most people would probably intuit M as 'million', but I think people still go with MM for avoidance of doubt.

It's the french system:

10M, ten thousand

10MM, ten million

10MMM, ten billion

here in sec. 3: https://www.druide.com/fr/enquetes/abréviation-de-million-et...

One would think that a metric country would adopt "k"/kilo as the indicator for thousands and "m"/mega for millions.

Still I suppose it's better than sussing out whether someone is British and using "long" billions.

(in most of the world, "billion" = 10 to the 9th power; in older British usage still sometime seen, "billion" = 10 to the 12th power, and 10^9 is "thousand million")

Well they're french, so M stands for Mille (one thousand). MM is just a thousand squared, and MMM is a thousand qubed.

Once you understand this system you might also understand why the long system makes more sense.

  1000                 = Mille     M
  1000 000             = Million   MM
  1000 000 000         = Milliard  MM M
  1000 000 000 000     = Billion   MM MM 
  1000 000 000 000 000 = Billiard  MM MM M

in the short system the billion (nine zeros) has nothing to do with two. while in the long system it represents double as many zeros as the million. a trillion has three times the Zeros. -ard suffix? add 3 zeros

As someone who used to speak decent French, you will never convince me that the language is rational with respect to numbers. The sometimes-decimal/sometimes-vigesimal thing still makes my head spin all these years later when I hear numbers being spoken in French.

"Still I suppose it's better than sussing out whether someone is British and using "long" billions."

Britain does not use "long" billions. Not since the 1970s or so. Now days, billion always means 1000 million.

Many other European languages, on the other hand, do use the long scale.

That’s correct. But when speaking or writing in English, the English conventions should be used.

And "M" dates back to the Roman number 1000.

Never knew this, never seen it in my line of work. Thanks for the clarification.

CPM in advertising. Cost per Mille (per one thousand views).

In the US, UK, and other English-speaking countries, "m" (note lower case) is used for million.

From the Guardian style guide:

"million - in copy use m for sums of money, units or inanimate objects: £10m, 45m tonnes of coal, 30m doses of vaccine; but million for people or animals: 1 million people, 23 million rabbits, etc; use m in headlines"

Presumably there is a considerable amount for reputational damage.

I hadn't considered it when I wrote my earlier comment, but I wouldn't be surprised to see O2 receive their own reputational damage invoices from their reseller networks (O2, Tesco, etc.) in relation to the problems.

No company in the world has a lower reputation than O2. It can't get lower

Only 60GB mobile data plans are £30 or more.

I doubt they would be compensating payment plans for devices.

O2's announced compensation is:

* Two days of an airtime contract (so e.g. if you just pay them £30 per month, that's about £2 in value for you)

* 10% discount on two types of pay-as-you-go payments with no specific limit but expect one to be announced.

The latter in particular you can expect to be heavily abused, because O2 needs the PR. I'd be astonished if that limit is any lower than £100, meaning they're giving away a £10 value.

I have a feeling you need to call and waste a lot of time, so most people won't bother... They won't lose near that much in real life.

No. O2 pay-monthly customers got a text message from them saying the credit will be automatically applied.

For what it's worth, I'm on a 20GB/month mobile data plan with O2 and pay £30/month.

Puts into perspective how many customers don't change their plans when they're able to...

Have you tried calling them up and asking for a better deal? I’m paying £16 a month for 40gb with BT, so surely o2 could give a better deal!

I suspect business customers might have more of a stick to beat O2 over the head with, rather than average people with a cell phone contract.

e.g. Transport for London's dot matrix displays could not show bus times for the 14 or so hours the network was down, apparently iPads issued to NHS staff to manage patient reports used O2 sim cards so that couldn't work either. Probably a whole range of small to medium enterprises affected too.

Our yodel driver reported they were struggling with both signing the packages as delivered and with drivers unable to cope without google maps.

Reply on news.ycombinator.com