Xfinity outage leaves tens of thousands in the Bay Area without internet | Hacker News Reader

Xfinity outage leaves tens of thousands in the Bay Area without internet

sfchronicle.com

73 pointsno_wizard72 comments

Their twitter feed was hilarious - individuals were reporting outages as part of a collective 'me too' and instead of Comcast saying there's a big issue with our network, they replied (probably with a bot) to every message asking to be be DM'd with the account details so they could look into it. To get to the outage map, the default path from their site (which was also down), you had to sign in to see that your very specific region had an issue. They really didn't want to show that (likely) millions of people were affected.

Hmmm. My tinfoil hat tells me that Comcast may be doing this to tie users Twitter accounts to their real names and addresses.

I mean, they can likely correlate that from their logs when you hit twitter and post if they really want to do that. Combine that with the info I bet Twitter sells on its users and it's got to be easy.

You don’t need a tin foil hat, that is what they are doing.

One nice thing about San Francisco is we have many great alternatives for ISPs. Comcast truly is terrible. Wave Broadband is an excellent competing cable Internet company. Sonic is the best ISP I have ever had or could even imagine; reliable symmetric gigabit fiber, super low latency (4ms to the edge, it's all glass), tech support that knows what they're doing. There's also MonkeyBrains for 100+Mbps fixed wireless. Not all of these are available everywhere, particularly across the whole Bay Area, but compared to most of America that's captive to a duopoly of mediocrity we're pretty well off.

Also Webpass (which got bought by Google fiber). It's only available in select buildings in SF but I've had good luck with them for the most part.

+1 webpass

I really wish I had other viable options. I live in SOMA and I have 1Gbps service from Comcast, which did go out last night.

My other options are DSL, so it's going to max out at 20Mbps or so.

I do have roof access, so I guess I could also see about MonkeyBrains.

All that said, during the outage, I just tethered to my iPhone on an AT&T unlimited plan, and was able to stream full quality video to my laptop. I think I speed tested it to about 80Mbps.

I have both monkeybrains and comcast, with two of us working from home it is worth it for peace of mind. The way the concrete flooring and stucco walls in the building work means tethering doesn't work very well but I would go with that otherwise. They are both pretty reliable and pretty fast overall, much faster than DSL. But have been a few times one goes down, or has intermittent packet loss. The monkeybrains setup we use goes over coax and it is compatible with comcast going over the same physical coax and a media splitter.

In the South Bay it's only AT&T DSL or Comcast cable.

I suppose Sonic is also an option that I would love to entertain, but they're just reselling AT&T DSL here.

If only we had municipal broadband, but the corps own the infrastructure and have long contracts with the cities :(

Meh, I'll vote to the contrary; Comcast works roughly 99.95% of the time for me. Obviously that's an approximation, but I rarely experience outages and I've been using them from two apartments in the South Bay for 10 years. The first few years they'd occasionally have local service outages around 2am but since then I rarely have outages of any kind and my speeds are great. Not as high as they say I do but supposedly sometimes higher when I do speed tests.

And as far as alternatives, in the Southbay it's very limited. I looked into this when I moved earlier this year and even Sonic only had DSL available where I live now. AT&T didn't even offer gigabit, and I'm not exactly in a bad spot other than that it's a suburb area (but I'm extremely close to a HQ for a FAANG).

Really miss having FiOS when I lived in NYC. AT&T has some dark fiber on the pole just outside my house but hasn't lit it up.

You should actually measure your network uptime. You’d be surprised how much maintenance happens at 3am. When people are up doing maintenance.

Make that 99.9%. There were two unscheduled multi-hour outages (each time a Friday afternoon) during day time of Comcast's service in the South Bay in as many years. Perfectly acceptable for a leisure offer. Those working from home these days who cannot rush into the office or take some time off, I suppose have a 2nd connection (e.g. 4G/5G).

Palo Alto is halfway between Google and Facebook. It doesn't have reliable fiber. Hell, the electric grid can barely be called reliable.

Sonic fiber is available by building, not neighborhood.

I'm glad Comcast is working well for you. The Bay Area Comcast seems reasonably well provisioned and doesn't suffer the massive evening slowdowns so much of the country does. Comcast still has a history of doing stupid things though, back to the days they injected RST packets to try to stop BitTorrent.

https://arstechnica.com/uncategorized/2007/11/eff-study-reve...

That's a relatively new "nice thing", though. Amongst a lot of techies it was actually a joke how bad SV and SFO had it with internet options until recently.

https://www.eff.org/deeplinks/2019/06/fcc-siding-landlords-a...

Monkeybrains has been servicing SF since 1998

My current apartment has Wave as one of the options, my experience is not that good and I know many Wave users are considering switching to comcast because comcast users in the same building report better uptime. We have one outage per month on average and Wave don't publish service status at all so it is almost impossible to know whether the outage is just my home router or issue in the area (you can call the support number but good luck with that when everyone else is calling the same number during an outage).

ISP monopolies are a huge problem and I find smaller ISPs tend to have better services. My previous apartment has a dedicated support team from a small ISP, and the tier 1 rep on the support line was capable of explaining the building's network topology and how they do natting.

This is true in the city. Unfortunately there aren't any options in the peninsula. When WFH started, I had 1000ms+ pings for over a month and had to have 2 alternate SIM tethering plans to keep working. Comcast acknowledged that they weren't able to handle the surge of traffic but still refused to refund the months long outage.

The few years our small business used Comcast Business were bad. There would be multi-hour outages every other month. It got to the point where I wrote up procedures so an office worker could log into our ASUS router and switch the gateway over to a smartphone hotspot. That got tiring, so I ended up upgrading our router to support automatic failover to an LTE device.

In contrast, AT&T DSL FTTN (Fiber to the Node) and Sonic DSL (not resold) have been rock solid. I am using AT&T for both home and business now and can't remember the last time there was an outage or issues with the modem. For me, a stable connection at 50mbps is preferable to unstable 1gig.

I would rather support Sonic, but they don't have fiber in my area and I had a bad experience with their resold AT&T FTTN service that was beyond their control. Their straight DSL product is solid, as long as you are close to the central office and can get a good speed.

Mendocino County, CA--My local Verizon data was also down for at least 10 minutes, likely overloaded due to the sudden outage, and local FM radio had no updates, as they do during events like wildfires or fiber optic cuts.

So for the second Comcast outage in a row, my first line of confirmation on the outage (as opposed to damaged house wiring or hardware or something) was ham radio. I turned on the HT, hit scan, and the first carrier found was a conversation about the outage, a couple cities north. That's always helpful just to get oriented. But funny because there ought to be a better information fallback now that internet service is so crucial. I couldn't even log into the Xfinity app on my phone.

Also, for the first time, I realized that texts using Google messenger's enhanced chat features may not go through at all in this situation, as one would expect normal texts to do. This was worrying because...do I need to disable this feature every time I go hiking now, if I want my texts to send?

I've had a horrible experience with RCS on Android to date. So much so that I fully disabled it again so I could actually send and receive messages. Very frustrating.

The HN headline says "Affecting US" but the article and all comments I've read suggest this is happening only in San Francisco.

Y'all must have some excellent broadband in California on average, wide-area Comcast outages are pretty common where I live, they certainly don't make the news.

Edit: Guess I spoke too soon, people are indeed reporting outages across the country.

I'm in BFE Indiana and to define it as "spotty" this morning would be... excessively polite.

The Midwest and east coast are also seeing outages

Is it that rare of an occurrence that it's worth reporting in the newspaper? Comcast goes down in my area a few times a year, every year, for the 20 years I've been handcuffed to them. It's not ideal, but I wouldn't describe it as a newsworthy event.

Same, but it’s rare that it’s widespread and protracted enough that spillover traffic affects wireless providers.

The difference is this happened simultaneously to several large populations. The Bay Area alone has a larger population than most states.

In addition to how widespread the outage is, it’s also taking them longer to fix it than any other outage I’ve ever had with Comcast.

I can’t help but wonder if this is the result of a cyber attack on Comcast’s infra.

North bay here. My Verizon cell service (calls + data) went down at the exact same time (10pm last night, for about 1-2 hours), and started working again at the same time Comcast came back online. Did that happen to anyone else? Felt like I was going crazy.

My GoogleFi service went down at the same time as Comcast. I am in Illinois. We had Comcast outage for almost 1 hour today around 7:15 am. I was on a work meeting. When my internet went down, I tried to get GoogleFi hotspot. Mobile data was not working. Then I tried to dial into the meeting. And it said my cell phone was not registered to network. At one point, the cell tower strength indicator in my mobile was showing a circle (no connection). Thankfully, now everything is back to normal.

This lists last night, but it seems as though there was a big issue this morning as well.

https://twitter.com/jgrahamc/status/1458087223983841284

Comcast/xfinity is horrible, last month during the atmospheric river my parents lost internet for 4 days. There was no update ever of when it would come back and their CS agent actually suggested that although they are working from home they should go use a public xfinity wifi spot... I suggested to him that he turn off his internet and go work at a public wifi point and tell me if that's a good solution

Don’t hassle CS agents in a situation like this. S/he was trying to help. Obviously the CS agent can’t go out and fix cables or reboot servers or do whatever needs to be done to fix it. But there is a chance that the customer will be better off knowing there is the hotspot option.

That’s not “hassling,” that’s “clarifying criticality.” The notion that we can’t even pass complaints along to support agents isn’t helpful. In my case when I lodge a complaint or turn down a solution it’s not meant as a personal attack. I legit want the company to know that they’re falling short and that I’m probably looking for alternatives.

Obviously don’t start getting angry, but if there’s a point of frustration please pass it along.

I spent some time time as a tier one agent, and will gladly confirm that any complaints given to us were forgotten by the time the call disconnected. If you're rude enough we might leave a note in your file as a warning to the next agent, but that's about as far as complaint handling ever went. Same goes for wanting a manager. Floor managers have no special tricks that they can use to fix problems, just nicer bedside manners.

Tier one CS is there to confirm where a fault lies, teach you basic troubleshooting, and send out techs. If the issue isn't on your premise the best we can do is check that outage for an ETR, though most of the time the crews won't establish one because that creates a deadline for them to meet.

Where did you work? Different businesses have different policies.

A third party doing support for Frontier Communications, among others. I wouldn't put too much faith in their practices being an outlier though.

That’s really only been my experience with telecoms. Other companies I’ve called in to have absolutely put me in touch with higher level managers who can actually resolve the problem. It depends a lot on what company you’re calling in to, but I’m not going to stop doing it just because it doesn’t change anything at EG Comcast. And I’m totally fine with you mentally erasing the conversation after I’m done.

'why don't you go work from a public wifi spot' is not a constructive or polite complaint. Just say, 'if you would pass along a customer complaint, here is what I'd like it to say: I need more reliable service to work from home; I may have to look for alternatives. I find it very frustrating. Thanks for sending that on for me.'

The human being at the other end of the phone is not your therapist or the object of your frustrations. It's the same with waitstaff. They have their own problems; they aren't their to be kicked when you are frustrated.

That's what Comcast told them to say. Don't abuse them; have some empathy - you have to deal with Comcast for a few minutes; they have to deal with it all day every day, and take the bullets from customers for Comcast's support policies and training.

> their CS agent actually suggested that although they are working from home they should go use a public xfinity wifi spot

That's a reasonable workaround, given that their home Internet is unavailable.

> have some empathy - you have to deal with Comcast for a few minutes; they have to deal with it all day every day, and take the bullets from customers for Comcast's support policies and training.

I spent 6 and a half years working for what was once the largest privately owned prime contractor that did work for Comcast.

I'll measure my empathy based on the fact that their continuing to agree to work that job is in fact enabling Comcast's abusive/neglectful behaviors towards customers, contractors, hell even non-customers can get caught in the crossfire.

I mean FFS, I made a point to live in areas so that I don't have to get Comcast and they -still- found a way to make my life a living hell a couple years ago. We're talking kafka-esque levels of "You can't speak to a human that will do anything but tell you to fill out this form with information you legally should not have to give." (Yes, I tried filling out the form omitting the information that was not legally required. They rejected it and then asked for even MORE.)

Sadly, the only thing that -did- work in stopping the harassment was to harass back. I would politely argue, ask them to consider a career that wasn't so soul-sucking, and point out at the start and end of every call how much time I had spent on the phone with Comcast, and the resulting number of dollars it was costing them to not just leave me the F alone.

I probably can never get Comcast service at this point, but at least they stopped violating my FDCPA rights and messing with my credit...

> I'll measure my empathy based on the fact that their continuing to agree to work that job is in fact enabling Comcast's abusive/neglectful behaviors towards customers, contractors, hell even non-customers can get caught in the crossfire.

Wow. I've seen people say that you shouldn't work for people like NSO, or national security agencies, or Facebook, but you should fall on your sword for Comcast customer service problems? There aren't many jobs left that meet your standards. I think we are taking rationalization way too far.

I experienced this outage last night, which doesn't seem to be a routing issue, because the cable modem can't establish connection with the headend(CMTS). but the the outage impacted area is big whole bay area and some other states, which seems to suggest issue happens on a central system, not just a few malfunctioned CMTS; my guess it is backend system related to managing cable modem or CMTS

One of my relatives recently lost IPv4 but not IPv6 connectivity through Comcast. She actually e-mailed me that her Internet was down. Of course GMail was the only site she used that supports IPv6, everything else was timing out. After power-cycling all the usual suspects it took a call to Comcast that actually needed escalation to get resolved.

First thing I think of is that poor engineer who committed the change that managed to bring down Comcast across the country. Oh to be a fly on the wall in that meeting.

The meeting where everyone will ignore all of the other forces at play that caused the outcome, and pretend that if only Joe or Mary had been better in some way, the whole disaster never would have happened.

Yeah, fuck that shit.

I'd say it's a bit early in the morning to get worked up over something that as of yet took place only in our minds.

Pretty common scenario in tech engineering.

Pretty common scenario in the universe. -FTFY

This is, at root, my conclusion as well.

I'd be willing to wager there are at least 100 HN members that had that exact meeting, with those exact dynamics, before this thread was even posted this morning.

At a previous job they sent everyone in the IT department to cause map training to improve root causes analysis. Mainly to get away from blame assigning.

Wm. Edwards Deming would agree with the actions of your previous employer.

Is this some sort of rolling outage? My service didn't skip a beat last night, but I had an outage in my neighborhood from roughly 8-9 AM CST this morning in the Chicago area. The cable modem couldn't get a DHCP lease.

At my office, also nearby, the outage was shorter; it's on a static IP.

I have been having strange issues with my Spectrum internet this morning.

Probably unrelated.

Power is out in some parts of the Pacific Northwest, there were storms last night

I realize that most people view broadband as a utility that should never be down, but are any of us really paying for even three nines of reliability, let alone five nines? Unlikely. Outages happen. This is why having a backup is important.

> This is why having a backup is important.

Which is something that's nearly impossible across most of the US, given the monopoly (or, if you are lucky, duopoly) in ISPs.

So does anyone know what happened?

Comcast is still open to this sets of issues..

https://media.ccc.de/v/32c3-7133-beyond_your_cable_modem

A email I sent just weeks ago..

________________________

>> ------- Forwarded Message -------- >> Subject: Serious DOCSIS maintenance network issue >> Date: Sat, 23 Oct 2021 21:09:16 -0700 >> From: Admin <admin@badmodems.com> >> To: xxxx@cablelabs.com, xxxx@cablelabs.com, xxx@cablelabs.com, xxx@cablelabs.com, xxx@cablelabs.com >> >> >> >> Hi all.. >> >> You guys hate me :( First it was Puma and the badmodems.com list and now this.. >> >> I am sorry to directly email you. No need to respond. Its OK, I understand its a legal thing. I wont email again. Sorry for this hassle. Sorry for a long read. >> >> There appears to be a very serious gap in your security best practices and policies that could result in a very widespread serious incident that could effect all DOCSIS systems worldwide and result in a worldwide incident. >> >> This appears to be from MSOs deploying horrendously bad security on the maintenance network. >> >> The issues are being discussed publicly. This thread begins with discussion of firmware and then turns to the maintenance network which appears to have little if any security implemented possibly because there is no modern published best practices for the maintenance network beyond something from the 1990's. https://www.dslreports.com/forum/r31122204-SB6190-Puma6-TCP-... >> >> The maintenance network, which controls all the devices on a DOCSIS network, is susceptible to attack. In fact its nearly criminally negligent in its lack of security and appears to be based on 1990's security protocols of mostly security thru obscurity. .. A subscriber on the LAN side can determine his address on the maintenance network and can ping ANY CPE on the network as long as they are on the same ISP. The CPE are not walled off from each other in any way. This could result in a VAST compromise of the entire MSO network nationwide from a 0-day worm that self spreads via the wide open maintenance network connecting all devices. . . ALL susceptible devices on your network, 10's of millions, could be taken over in hours with a self spreading worm with a nearly impossible task of clean up and maybe a week of complete ISP downtime. This would also result in the largest loss of subscribers in history for cable as people flee to DSL and 5G that day trying to get internet. You would need new firmware for every device that addresses the issue, and getting new firmware will take weeks. All the susceptible CPE might be bricked with no hope of recovery once taken over. The current security practices are inadequacy. The news coverage would be devastating. Each modem/router could attack the subscriber side and scrape data and files. On the ISP side it would lock out all maintenance access and recovery of the devices, and the whole network, nearly impossible. It would setup a serious botnet - possibly the largest ever created when combined with the other top world wide ISPs. It might even result in a Ransom ware attack on a massive scale with all the CPE locked out from the ISP. A silent malware could spread stealthy and then sit on CPE and attack the subscribers quietly by doing fake DNS and even MiM attacks. This could already be the case. A botnet of CPE would be incredibly powerful >> >> This wide open gap appears to exist in most ISPs. So it is a CableLabs lack of proper security vision to keep up with modern threats by doing best practices for the maintenance network seems to be the main issue. 10G offers micronets and SDN containment of LAN devices,,, yet the ISP has nothing like it to protect its own network and its subscribers. >> >> Each ISP will need to do a 3rd party security audit and pentest of all the MSO's maintenance networks and secure them. The kinda emergency level, possible fairly easy temp fix is simple. Isolate each piece of CPE. Right now all CPE can see each other and spread worms. Simply doing a config change could wall off each device with NO downside. This might be able to be implemented maybe in a day. This alone would reduce the issue to nearly zero. BLocking access to the maintenance network from the subscriber is also key and most likely easy. MSOs REALLY need to do this and because these discussions are going on now, badguys could be reading, so RIGHT NOW is the time to secure MSO networks BEFORE a incident occurs. >> >> There may be simple quick solutions to avoid this doomsday scenario.. Make sure you read up to the current postings. https://www.dslreports.com/forum/r31122204-SB6190-Puma6-TCP-... >> >> I will be following up to be sure you got this message. >> >> You can contact me for any further details or respond to this email. >> >> I am the guy who found the Puma issue. So you guys know I can be persistent and noisy. I would really like to hear that CableLabs is going to pursue a whole new approach to device security on the maintenance network including RAPID firmware deployment. EVERYBODY wins.. >> >> Sorry for blasting the email. Sorry to start your monday kinda ruff. Think of it as a cool new feature. >> >> I have contacted all the top 10 MSOs and sent reports to the security teams. They are the guys who made this mess, but, they need a good best practice to follow and that does not seem to be there. >> >> Gone are the days of junk boxes with poor CPUs. MSOs are dropping POWERFUL devices with lots of RAM and Flash. They run Microsoft or Linux. They are connected to a massive bandwidth pipe. It looks possible to take over whole ISPs. These are prime targets no one has noticed yet apparently. Gone are the days of old.. These are high value targets and a bot net of incredible scale... Its time for a top down new approach to firmware and device security.. >> >> Of course none of my doomsday scenarios most likely will ever happen.. And most likely everything is fine.. BUT MSO's can't just keep these maintenance networks so 1990s sloppy. >> >> IMHO.. >> >> xxxxxx xxxxxxxxxx >> >> >> >

> “This comcast outage is a great reminder of why it's still important to download and own physical copies of music, movies, etc.,” @krakinov tweeted.

This is a bizarrely overdramatic response to one provider being down for an hour, and I can't imagine why the author felt the need to include it, other than to further dramatize their own dramatization of a rather innocuous situation.

> further dramatize their own dramatization of a rather innocuous situation.

That's exactly why they included it. It's the internet equivalent of when journalists go out and interview random pedestrians to get their reactions/commentary to some event.

They want a soundbite that will stick in your mind after reading/viewing the news segment, regardless of how useful the commentary really is.

As long as most news is for-profit, and therefore their incentives lie primarily in maximizing engagement/ratings/ad revenue, they will continue to optimize for those things with different iterations of what is essentially clickbait.

Even here on the relatively intellectual HN, I regularly see articles posted that have very clickbaity and misleading titles.

Now journalists don't go out and interview people for those reactions they just troll twitter and publish articles which are essentially multiple tweets.

How is someone's Twitter comment evidence for a rant against journalists? Do you see such a problem in the OP?

"This power outage is a great reminder of why it's still important to maintain and own wood stoves, kerosene lanterns, candles, etc."

Next he's going to talk about keeping a horse in a stable out back in case the gas station runs out of fuel.

I don't know about you, but if I lived in a place where the power goes off I'd certainly have good torches and non-electric heating.

(Of course if internet goes down my phone has tons of spotify tracks downloaded onto it, so I'm sure I'll cope)

Sure, but light and heat are pretty fundamental to meeting a basic level of comfort. Whether or not I can listen to my favorite album without internet access isn't.

> Whether or not I can listen to my favorite album without internet access isn't.

What about being able to work?

This comment thread is about someone specifically calling out having an offline copy of music and movies.

Reply on news.ycombinator.com