What happened with your Internet last night, and what we are doing about it | Hacker News Reader

What happened with your Internet last night, and what we are doing about it

corporate.comcast.com

41 pointshw43 comments

The trouble description seems a bit off. They had a backbone switch in San Jose fail (according to mtr), but it only affected DNS depending on your routing. In my case, 8.8.8.8 and 8.8.4.4 were both on the working side of the switch, but a lot of other sites and systems weren't -- even when accessing by IP. So it wasn't really a DNS issue.

I couldn't ping Hurricane Electric servers, but Slashdot worked OK for example.

The frustrating thing was that I was going through a Sacramento hop first but still couldn't reach systems in Reno -- I thought there was fiber all the way up the 80 corridor, but apparently most of my Comcast traffic has to go to San Jose before it can proceed East.

    but a lot of other sites and systems weren't -- even when
    accessing by IP. So it wasn't really a DNS issue.

That's just an intermittent issue that Comcast customers deal with--I experienced that in Indiana a couple of months ago.

I believe that it relates to them re-jiggering routers such that the appropriate default gateway for your modem changes, but because your modem is already up, it doesn't get notified of the change. So the modem doesn't send the message to the correct router, and the router it sends it to may or may not know how to get it to the destination.

I typically have logs streaming through terminal windows from several servers and multitask online -- I notice even brief transient network issues. FWIW what you're describing hasn't been a problem for me (though I don't doubt it is for lots of customers), and the San Jose outage was definitely out of the ordinary.

Anyway, I wish Sacramento was a bigger network hub, is all.

What a generous window of time...

    This $5 credit offer is being extended to our customers with active XFINITY Internet service in
    California, Washington State, and Tucson, Arizona who were affected by the issue with our DNS
    servers that was first reported on 6/1/15. The form above will be available until 6/11/15

I agree... They should just add that credit automatically...

Seriously. They know which customers are in the affected areas because they have their billing addresses. Just credit them $5 on their next bill.

Further:

  Please allow 4 to 6 weeks for the credit to be posted to your Comcast account.

So, they refund you $5 in Comcast credit that will eventually come out of your next bill?

What if this incident caused you to switch to another ISP prior to said bill. Based on my experiences good luck getting that $5.

"Switch to another ISP"? That's a funny joke.

Comcast exists to prove that government isn't always less efficient at providing service than the private sector.

The rule of thumb is that government tends to be less efficient than private markets. The residential ISP "market" isn't much of a market in lots of places. Comcast tends to operate as a monopoly or near-monopoly.

I'd have to wonder at the product we'd get if we swapped out Comcast and let the Federal Government take over their business.

I think it would cost net twice as much (total expenditures out of the consumer pocket via taxation, the plans wouldn't cost more), be about as reliable, about as fast, lose lots of money and still be completely unaccountable to its customers. The system / infrastructure upgrades would be slow, bureaucratic, always vastly over-budget, and a constant funding fight in Congress.

(I say this understanding fully that few advocate for a federal system, but rather local muni connectivity)

Comcast doesn't face typical market competition; it is uncommon for it's customers to have good alternatives. That's not a good representation of the efficiency and quality the private sector produces.

> That's not a good representation of the efficiency and quality the private sector produces.

It's a good representation of the danger in the common quasi-religious belief that hasty privatization necessarily entails a marketplace with healthy competition.

Surely no true scotsman....

I think it's fair to say that taking any single company and holding it as evidence of what the private sector produces is flawed, but Comcast is a particularly flawed example. Likewise with any single government entity.

There's also this part too:

  This request form is for our residential customers only.

I have a business account, and my connection was affected...

Business is same as residential, they just show up at your house first.

That's nice. I figured out what was going on (though I was very frustrated by that point) and switched my DNS servers to Google's public DNS servers (8.8.8.8 & 8.8.4.4) which worked fine. I figured that calling would be a waste of both my time and Comcast's - the people on the other end of the phone have no idea how long a fix is going to take other than 'as soon as possible'.

The straightforward explanation and humble acceptance of responsibility were a refreshing surprise, however. A $5 credit is not that big a deal but it is good to see a firm putting its money where its mouth is instead of trying to deflect or minimize the disruption to customers.

So Google runs a public DNS service too? First time hearing this. I was annoyed with the constant DNS outages but did not do anything.

They have since 2009.

https://developers.google.com/speed/public-dns/docs/using

8.8.8.8 man, what else do you ping to see if the internet is working?

For nostalgia, sometimes 4.4.4.4 ;)

Edit: Make that 4.2.2.2 (it's been a while).

www.something.com usually!

127.0.0.1, but I do get quite a few false positives.

My comcast service was really twitchy last night. Page load timeouts, ssh disconnects... but I'm not in the western US and I don't use their DNS (OpenDNS user for years).

I've set up a separate location in my network settings on OSX that points to the Google DNS servers. It's come in handy a handful of times.

I wish there were more consumer routers that came with Google DNS as a default backup.

It would be nice if these regulated monopolies would actually charge a decent flat price instead of promo rates that then jack up 6, 12 or 24 months later.

This $5 is a different percentage of about 6 different price levels for Internet.

How generous of them to offer this for anyone who happens to learn about it in the limited time window they are offering it for... (/s)

Also: $5 is half of the $10/month modem rental fee (that I assume is commonly paid by a lot of their customers who don't know any better) on a ~$70 modem (probably much cheaper at the scale they buy them at) that has probably been paid for in rental fees 5 times over already.

If my recent experiences with them in regards to DNS is any indicator, their DNS infrastructure is a colossal joke. 4 days to get a reverse DNS entry done? Really?

I wont even mention the typos in some of their SOA records.

I'm not sure if $5 adequately compensates for the productivity (and in some cases entertainment) loss encountered. I guess that equates to roughly a day's credit depending on what your plan is like. Plus I don't see why they require us to file a compensation claim, when they could just credit every account that was in the affected areas.

I wonder if they would be compensating more if the Internet and TV went down during a big event like the NBA Finals yesterday.

As a consumer service is there even an SLA!? I mean stuff goes wrong, if your home internet connection is so important then multi-home leased lines is what you need..

Step outside. Do some jumping jacks. Talk to your kids.

You pay $150 for Comcast Internet?

Obviously requiring an opt in cuts the payout by 99%.

Technically I pay $224/mo for Comcast Internet, but apparently (home) business customers aren't eligible...

NOW it makes sense.

I was writing up a tutorial which involves working with subdomains. I was QAing the tutorial by following the tutorial along but the DNS changes wouldn't work. I know it worked before but not this time. And meanwhile I was able to visit the websites that I usually visit.

After seeing the Comcast letter, I'm concluding that the DNS changes weren't working because of Comcast's issue. I was still able to visit websites because my home DNS server had cached DNS names.

Oh well.

>We are also building a Web site that impacted customers can visit to receive their credit. We will update this post with a link to that site as soon as it is available and will share the link on Twitter through our customer support handle @comcastcares.

If you need to use a social media account name that says "company name cares" then its obvious that most customers think you don't care.

My home comcast connection in SF started being extremely flaky last week and I switched over to 4.2.2.2 etc and haven't looked back.

Level 3 isn't cool with non-customers using their DNS server.

Google operates 8.8.8.8 and 8.8.4.4 that are for public use.

Level3's VP disagrees:

http://blog.level3.com/level-3-network/a-flawed-study-of-cdn...

"Open DNS is provided by companies like OpenDNS, Google and Level 3. You can use it wherever you are on the Internet with no restrictions or authentication required."

Huh, that's interesting. As an organization, there's no public info on the availability of those servers.

Last year they showed ads for failed DNS requests for non-customers. http://james.bertelson.me/blog/2014/01/level-3-are-now-hijac...

So either business class customers were not effected, or are not eligible - personally. I didn't notice a DNS outage, I run my own DNS locally.

Reply on news.ycombinator.com