BT fault hits broadband users and banks | Hacker News Reader

BT fault hits broadband users and banks

bbc.co.uk

90 points0xbadf00d65 comments

It's a power outage in Telecity Harbour Exchange, from the LINX mailing list this morning:

08:12

- We are currently experiencing an outage impacting some of our equipment located at Telecity/Equinix Harbour Exchange in London. It is impacting both the Juniper and Extreme LANs.

- We are still investigating to get further information, but we have lost all management access to the devices at the moment.

08:45

- We have had confirmation from Telecity/Equinix that there was indeed a partial loss of power impacting one of the power feeds.

- Power failed at 07:55 BST and was restored at 08:17 BST for the majority of our equipment.

- Unfortunately one of our edge routers (edge4-tch) on the Juniper LAN is still down at the moment.

09:33

- Power to edge4-tch has also been restored now. The router has been back up since 9:15 BST.

- All member services should be restored.

- We have an engineer on site and are working with Telecity/Equinix to understand why we lost all power to that device.

- We are also waiting for a full report from Telecity/Equinix about the cause of the power failure.

---

We don't peer with BT so ou traffic was OK but we did notice loss of service from AWS to BTNet

Great update thanks.

Perhaps I am being naive, but I thought data centers had back-up power to cover this exact problem? I.e. battery power until the generators kick in... or is this how it used to be done and these days no one bothers and just relies on fail-over to other sites????

despite best efforts, sometimes the generators don't kick in. sometimes not all of them kick in. sometimes one ATS in one feed in one section that just happens to house one half of an important switch chassis doesn't work as it did during tests last quarter.

True. But Data centers usually have an A and B power systems independent of each other for redundancy . Equipment is usually connected to both. The failure of either should not cause an outage. (Unless some one is stupid enough to connect core network equipment to only A or B power)

Sometimes only A power or B power is delivered to the rack (otherwise you could receive different phases).

Power failures also occur sometimes when the issue is on the wrong side of where power is distributed - we've lost power to our servers when maintenance has gone wrong for example.

From my point of view if you end up with mission critical equipment connected only to A or B power then its engineered wrong.

Very true, but sometimes this happens at fault of the facility, not the affected company. I've been in a facility experiencing widespread power issues because customers were incorrectly delivered A+A or B+B power, instead of A+B.

This is why I'm a proponent battery floated of DC Power for mission critical systems.

Exactly. You can only test for these scenarios to a certain extent and even when everything works as expected during tests, things still tend to break during production when shit hits the fan.

Like you, I'm surprised.

I thought major data centres had redundant power from the grid too (not just battery and generator), so the UPS smooths the power transition as the switching gear fails over to the alternate grid connection; then generators come up if that fails too? Is that considered over-engineering?

They usually do, but these only protect from grid failures. HEX certainly does have battery + generators (it's one of the most important DC's in the country).

Components inside the DC should be redundant, but the customer can decide whether to have dual power feeds from redundant areas of the DC provided to their rack/cage.

Even with all this, issues do happen, Telecity had a very high profile and prolonged issue at the end of last year that affected AWS & Azure amongst others: http://www.theregister.co.uk/2015/11/18/telecity_outage_fix_...

Perhaps a more serious point is that a single power failure (in one data center) can have such a wide impact.

The best thing about being BT is that you own everything so you don't have to justify anything to anyone. Since half of the lords have long term shares in the monopoly the moment that it went private, then your back is covered as far as law is concerned so long as you pay your dividends. You can just laugh as the customer goes somewhere else, because you will still be picking up 75% of the broadband bill.

I also love how BT pays out so much to shareholders, and the public purse bankrolls the infrastructure, then everyone pays BT wholesale. What an outstanding deal.

BT is corrupt and I would like to see it fragmented.

BT doesn't own Telecity I suspect that if they did it wouldn't have happened.

And BT pays out around 3% dividend to its owners which is not excessive a lot of BT's detractors have an axe to grind (Sky, Murdoch and Virgin) and want to cherry pick customers and to hell with the USO.

sounds a bit like Deutsche Telekom.

I find the Talktalk chairman's quoted comments hilarious, though I assume they're not specifically in relation to this outage. If it were Talktalk's network spending would drop about 90% and complaints rise a similar amount. It's not like TT would be unaffected by a Telecity outage. I don;t think TT ever cared what their customers think, just reducing prices and service to the bone.

OpenReach own all the ADSL wiring - that's what TT have to buy from them. Most consumer ISPs are quite 'virtual' and dependent on BT for actual provisioning.

The manager of a tiny geek-friendly ISP, Andrews&Arnold, frequently complains about this on his blog: http://www.revk.uk/

Well they own exchange to home. Since unbundling it's a bit fuzzy as most of them have their DSLAM in the exchange and pay rents for space in exchange and run virtual over BT's backbone and so forth. But at least it freed up a lot more competition. Hence Sky asking for Fibre cabinet unbundling recently.

If it was A&A or Zen's chairman quoted there'd have been no need for snark. ;)

> Talktalk chairman's

Baroness Harding of Winscombe -Dido Harding- is Chief Exec of TalkTalk.

Sir Charles W. Dunstone is the non-exec chairman of talktalk.

I'm not a TalkTalk retail customer, but I do use their backhaul through another ISP, and it seems very reliable.

I expect the wholesale service is a very different animal to retail. I'd expect them to lose custom very quickly if they weren't reliability competitive. The customer is usually going to blame whoever they pay the adsl bill to, even when they're not the cause.

TT often topped the retail broadband complaints charts. Though BT "won" the last one I saw with the EE merger.

Graphs here: http://www.ukinternetreport.co.uk/?p=d&id=215

UKNOF's mailing list is private so it's hard to get a handle on what's going on, but from what I've heard, a DC went down and BT's infrastructure hasn't recovered.

There isn't any insightful discussion going on. Suprisingly. So you are not missing out on any info.

There's not a public archive (AFAIK) but I don't think the mailing list is actually private.

Level 3 outage i believe

Apparently down to Telicity Group DC having power failure which BT heavily rely on

Which DC, there are many.

Two London ones I can think of is north of the city (Oliver's yard) and docklands.

Arstechnica says Telehouse North Docklands http://arstechnica.co.uk/business/2016/07/bt-isps-telehouse-...

We've updated the story - did some more investigation. Definitely looks like it was Telecity.

We run one the most popular speed test site in the UK at http://www.broadbandspeedchecker.co.uk . We are getting 20x more people testing today than usual. Also, because Ookla is down.

It'd be nice if we got email notifications about this. I wasted time trying to work out what was going on (and if the issue was in our local network).

You want to them to email you to tell you your internet service is down? Er ...

Funnily enough I can get emails on my mobile, even when my landline is down..

Not sure if it is a complete outage, I'm having issues with many sites, but HN works fine - which usually points to a DNS issue cough

I see the point you're making (and I did laugh) but a lot of us get e-mails on our phones and would still be able to pick them up if our ISP went down.

Most people have redundancy. Computer has wifi, phone has LTE. Usually different providers.

I would love an email thst my internet service is down. Or text.

Incidentally BT also has infrastructure or delivering SMS to landline voice numbers via text to speech - or at least had as of a couple of years ago - so as lon as it's the broadband service and not the end user lines that were down they'd still be able to deliver it.

Many of the niche ISPs such as AAISP provide line status notifications.

Lots of people get email on their phones these days via a mobile data connection. Our DSL connection isn't 'down' - just really, really slow.

The mobile connection still goes through an ISP...

Yes, because email can be received on mobile phone which is outside of BT

The infuriating this is that whenever this happens at BT, they say nothing about the cause and nothing about how they will prevent recurrence. As a BT Infinity customer, I've had a total loss of internet several times over the last couple of years and never any feedback from BT afterwards.

They seem like one of those obnoxious organisations that think if they provide any information then it will only invite a shitstorm. Unfortunately too many people still chose them as a provider. I nearly. Nearly switched to them last month because of their prices and BT Sport package. But fortunately I'm stuck on a 12 month contract with IDNet. Shame.

As a colleague of mine used to say when I worked there: "It's a good thing we're not a communications company."

My BT Infinity home broadband in south London is affected. 'mtr' shows any traffic via the BT-owned IP address 31.55.187.177 gets 88% packet loss when it hits that router. Web sites using other routes are ok.

And again the next day: https://news.ycombinator.com/item?id=12135794

I noticed this morning that HTTPS was failing but HTTP was fine, but it "fixed itself" after ten minutes. No idea if this was related.

Had issues yesterday where BT was complaining that my machine was using non BT DNS (all our machines at home use google's 8.8.8.8 / 8.8.4.4)

Slightly concerning since we never enabled parental controls (Which the message implied was the reason it wanted us to use BT DNS) and now hearing that somehow only https traffic was affected by an issue.

As an aside, I've found BT's DNS to be very flaky; sometimes down (or unreachable at least) for many hours at a time. I switched to google's and never had any problems since.

This sounds very troubling. I hope it was just a coincidence.

Is there a reason there are so many Data Centres in Docklands? Is there something about that area that makes it great for datacentres?

Most investment banks are based there (Canary Wharf), they pay a lot of money for low latency.

For the same reason why datacentres are common at any former industrial area: vast amount of electricity.

Huh, so that's why my internet connection was timing out on a lot of sites earlier today.

Their customers should rightly be outaged.

This is probably the reason why linux.dropbox.com was down for brief time today

Out of all the sites you could have mentioned why did you think that (the deb repo for Dropbox's packages) was pertinent to this story?

Because it was. This and ubuntu repo was not working on my PC during BT outage, other sites worked. After BT sorted it out I didn't have any problems. I asumme that it was the issue with some BT connection between me and dropbox server

Might be a better organized coup attempt from the remain supporters :)

Middle and upper class folks cutting off their Netflix and Spotify entertainment while also leaving them unable to tweet incessantly about it. I don't think so.

Living in England, you really need a downtime strategy for your internet connection. While it's seldom a complete country-wide thing like this one, you can still expect your broadband to drop out for up to an hour at a time at least once every day.

BT is just that incapable of providing DSL to residences.

I live in a tiny village in rural France half the time, and never really think about the internet connection. Just like you never worry that water won't come out of the kitchen sink when you turn it on.

But then I come back to England for the summer and there at the front of my mind several times a day is the Internet connection. Why is it super slow all of a sudden? Hey, it dropped again. I hope it comes back soon because I need to check stuff in. I have a personal hotspot set up on my phone that gets used within my house on a weekly basis. There's an OpenReach van parked out back a couple times a year fixing yet another fault on our line.

Nowhere else I've lived has it been anywhere near this bad. I look forward to the day where BT has some actual competition.

I think to an extent it depends where you are. While I'm far from BT's biggest fan, the actual internet connection they provide is pretty reliable here.

The piece of crap router on the other hand needs rebooting to bring the wireless networking back up every few days, and has absolutely no logging to diagnose why that's the case.

I've been with BT for a very long time (5+ years) and other than during unseasonably violent storms this is the first time that I've suffered any kind of outage at all. Not only that but I'm in the middle of nowhere and still get fibre (to cab at least)

Then you're been very lucky.

http://www.theregister.co.uk/2016/02/03/bt_blames_faulty_rou...

http://www.theregister.co.uk/2015/11/17/bt_outage_three_days...

http://www.theregister.co.uk/2015/07/29/bt_broadband_in_broa...

http://www.theregister.co.uk/2014/12/12/bt_infinity_working_...

http://www.theregister.co.uk/2014/06/30/bt_major_dns_outage/

http://www.theregister.co.uk/2014/02/25/bt_cable_theft/

And that's in just over two years.

Reply on news.ycombinator.com