East Coast Internet Outage
Anyone else noticing a widespread internet outage on the East Coast of the US?
There is a fiber cut in Brooklyn. We have no ETR, as of yet. You can use the MY Fios app for updates. *EAG
https://www.wsj.com/articles/verizon-internet-outage-disrupt...
> The company hadn’t pinned down the source of the failure as of Tuesday afternoon.
Although my packet loss to my IPv6 tunnel provider just went to zero, so that's a good sign here (Manhattan).
How are the fiber optics being cut? Are there no procedures for this or is there some challenges how the infra is setup today?
You can't realistically prevent this with additional armoring, because it's quite a different failure to someone drilling in the wrong place - the forces are a lot higher, or heavy machinery is involved etc. (see also: the rate at which workers go through water mains in footpaths).
One idea, borrowed from some chainsaw safety pants, would be to wrap the cable in a fiber web sheath that tangles in whatever cutting equipment
This spun threads reminds me of nature. Spiders, silk works spinning web. Wonder if that's where the inspiration came from.
Simple! Bury a network cable and wait for the backhoe to show up.
Backhoes are VERY strong and will easily go through gas pipe let alone anything that might protect fiber. And if the fiber is not where the plans say it is, even calling 811 won't save you.
I know newer infrastructure is more diligently recorded, but there is a TON of old crap in the ground ready to be ripped up.
Everything else is 100% correct!
I once hit a gas line. With a jackhammer. 8” below the surface. That should never happen, but did.
Level 3 at one point stated 17% of all fiber cuts were from squirrels... https://web.archive.org/web/20151213214019/http://blog.level...
The lines are supposed to be known/marked to some extent but you can't avoid the human factor.
In our worst case the same city cut the fiber line in multiple places over the course of a few weeks. Our crews would finish patching one fiber break only for the same (city) crew to cut it again a few hours later.
I wasn't on the ground so I don't know exactly how they managed it, but it was a real headache for us to manage on the customer end.
The dig safe people can and do mark them, but people tend to do stupid things.
Some highways have fiber rights of way too. You can see fiber conduit on bridges on the mass pike, for example.
Where are you located (roughly)? Having assisted in remodeling of 7 homes in the US (IL, WI, CA) built between 1880 and 2005 I haven't seen this.
If the cable is passing through a stud less than 1.25" from either face of the stud (giving you ~1.75" once drywall is installed), then a metal plate has to be installed to stop someone trying to hang a picture or shelf or something innocuous up from driving a screw into the cable.
As for fiber cuts, one case I'm aware of involved a large diameter concrete drill/hole saw with a diamond-tipped blade. Not much you can do to armor against that.
There's always the person with a rented ditch digger or over confidence "Well I dug out here last year and didn't hit anything, so I can dig again" who doesn't call or notice the trace cable, but overall that service works pretty well at preventing disruptions.
Network never went down completely, bitrate just took a nosedive.
Secondly, ZERO countries in the world have non-fragile infrastructure. America's infrastructure is better than most countries and definitely above average even among developed countries, except public transportation.
This will never be possible, even if America does nothing 'wrong' against anyone else. As long as America (flaws and all) is a better place to live than other countries, and can be used as an example of how things can be better (though not perfect), and so will be a target of powers that don't want their own people thinking about improving things.
* https://en.wikipedia.org/wiki/The_Road_to_Unfreedom
* https://www.theguardian.com/books/2018/apr/15/the-road-to-un...
There are also plenty of bullies out there, who are only checked by American power, and wouldn't mind the US being more isolationist.
There are all sorts of problems of how America interacts with the world, but there are many other countries who would do worse things IMHO. America simply existing generates enemies.
We should try to make enemies only when attacked. Not make enemies out of adversaries.
There are innocent people who have done nothing wrong, and yet other folks want them wiped out of existence.
I'm sure we can all think of examples from history where this is true, both the recent and distance past.
This comment is not based in reality. Does Australia have many enemies? Does South Korea? Does Japan? Taiwan?
And of course the fact that for the most part those most impacted by these outages aren't the ones making the decisions about what money to spend in the first place.
I have no meaningful influence on my ISP's network backbone. I don't even have a real choice in ISP so it's not like I can go to someone else, and they know it.
I currently have 500mbit/sec service from my cable company and the only other options are ~50mbit/sec 4G or Frontier DSL that claims 24mbit/sec while rarely syncing beyond 8. Those aren't actual alternatives. If I want to have a connection that can properly support a household full of gamers and streamers, I'm stuck with this unless I want to add another zero or two to my monthly bill for a dedicated fiber line.
This is the best case scenario for the majority of the US. One cable company that's mostly decent, one shitty DSL provider, and a few wireless services that have very limited spectrum and data caps. Almost no one in this country has true competition in wired broadband unless they have a municipal fiber service or are lucky enough to be in one of the few cities that Google dumped money in to.
FiOS customer in DC Metro.
Edit: Also, posts without URLs get a downweight by default. I've turned that off too, and merged the other threads into this one, since it was the first one.
An NLP/Machine learning model would be a great way! Probably also taking in other features as input such as the number of comments per second, and how deeply nested those comments are.
Let's build one!
Why? There do not appear to have been any posts remotely resembling a flamewar in the comments.
Any theories out there to explain this phenomenon? Fascinating.
(My theory is a network partition, where my roommate’s IP addresses happen to be in regions that are on “our side” of the partition, whereas my IP Addresses are on the “other”, unreachable side).
In any case, it looks like it’s ended for us. The window for gathering evidence slides closed...
It’ll be interesting to read the post mortem on this one.
Too much traffic is concentrated at the major carriers which carry a large amount of traffic over relatively little fiber (cause its cheaper) and when that fiber disappears we lose a massive amount of capacity...
The major internet connections are like superhighways, with major central routers acting like exchanges. If a major superhighway had an accident blocking traffic, there would be personnel stationed at exchanges to help re-route the traffic, though that often occurs too late to prevent a bunch of backups. At any rate, people using Google Maps, Waze, etc. would see the accident and would likely be redirected to alternate routes that would normally be much slower - and would also be slowed down by the influx of higher than normal traffic.
The fact that the internet still worked albeit much slower than usual leads me to believe that something similar happened. While normally my internet traffic might go through a major hub in NYC and then onward to various carriers and hosts, etc. it had to be re-routed around that hub that is normally sub-optimal, and also ill-equipped to handle the spillover. (And in some cases, bandwidth was saturated, and failed altogether for some users for certain usage.)
Then they get to dig it out better and someone gets to get down in the hole and figure out which piece of fiber needs to be spliced to which other one.
Because of DWDM, some of those individual fibers may each be carrying 64 channels that in turn each carry 400gbit/sec for different providers/services/etc.
It is still a lot of work, and so far not automated.
You should take some time to understand the physical layer of infrastructure. There will always be work to do in meet-me-rooms!
Source: I have no idea.
So imagine if you cut a Cat6 cable in half, and then spliced the individual wires back together without regard to which wires you connected. Very likely, Pair 1 negative is connected Pair 3 negative, Pair 2 has its polarity reversed, etc, and so the network link never comes online.
At the individual device level, fiber cables are connected in pairs, in a crossover fashion. Device A Tx connects to Device B Rx, and vice versa. So with a random splicing, you would end up with pairs getting crossed or broken up, invalidating the connections and labeling of every patch panel downstream, making for yet more work than the initial work of splicing the cut bundle.
1. Not impugning your intelligence, knowledge, or character here, some people never have because they've never needed or wanted to, and that's totally okay.
Note: I am not a fiber technician nor do I work for a large ISP.
Ironically things recovered within 30 seconds me of messaging our systems team to ask if the development VPN was having an outage. That's like a magic rule of internet outages, isn't it? Nothing works until you try to show someone else the issue, then it all works.
In the entire history of the Bell System, no electromechanical central office was ever down for more than 30 minutes for any reason than a natural disaster or, one occasion, a major file. That record has not been maintained in the Internet era.
Also, while my Internet may have outages from time to time, I can't recall the last time I couldn't use my cell phone other than being in a bad signal area. For voice, cellular is definitely the way to go as capacity is easily added by adding new towers; not something easy to do with landlines.
Your cable modem (which has a CPU and does high-speed signaling) or your FIOS box is much more complicated than an analog phone.
The carrier-grade router on the other end of your cable modem is really just a bigger version of routers you buy off the shelf or for enterprises. It's literally routing IP packets in the same way.
Whereas you have to have a whole switching system at the other end of an analog voice CO. (Or an operator making physical connections if we're still in the 1800's). The best you can do for yourself in that world today is get an PBX system like an old NEC NEAX box or similar, but you're still not going to be on the same level as the CO.
Are these systems even comparable? Today /= yesterday phone service....
and that AS42861 isn’t clearly defined under Roblex
edit: so does https://www.peeringdb.com/net/2258
Based on other comments we probably just didn't notice the big actual outage that happened all at once.
Doesn't look like it's isolated to any one provider, or any one site. https://downdetector.com is showing many issues across many platforms.
Less pessimistically, it's possible that the regions that're affected have natural "choke points" for infrastructure, either moving through the mountains or along highways, etc., which of course leads to a single-point-of-failure, though it may be unlikely.
In principle, you can have systems that very quickly route around damage like this, but in practice it happens so rarely that it's not worth it for consumer ISPs. I think it should exist, but the ISP space is so burdened with red tape that the only way anyone could compete with it was literally to start a rocket ship company and launch a bunch of satellites.
https://arstechnica.com/information-technology/2021/01/jared...
> Mauch has been charging $65 a month for symmetrical 50Mbps service, $75 for 250Mbps, and $99 for 500Mbps,
That's less than Comcast consumer rates in Baltimore.
edit: as of 13:20 (GMT-5) it appears to be resolved.
Aside: Comcast, wtf? 1.2 TB limit per month? Seriously if anyone works at Comcast know that I hate your company with an overwhelming passion.
edit: hitting 48-50 mbps again today, sweet
I reliably get 900mb down and 900mb up with Verizon in DC.
(I'd expect that only in the past year or so has upstream been a lot more heavily utilized, what with massively increased work-from-home due to the pandemic. Before then, the primary stressors of the network were things like streaming 4k60 video.
And I'll be very surprised if any ISPs thought torrent uploads to be a use case worth optimizing for.)
Wife wanted to live out here. So we traded my being able to work easily for amazing views and lots of land. Not entirely unhappy with the trade, but there are days when I think about just renting an office in town. (30 minutes away).
Unfortunately, the cable company usually offers some ludicrous 500mbps (burst) download / 5mbps upload connection that people are willing to accept for cheaper, so if the phone company comes in and pays to install fiber and charge accordingly for the new fiber runs, insufficient people would switch to the fiber to make the numbers work since they are okay with the shitty cable company connection.
https://todayamerican.medium.com/the-definitive-guide-on-how...
>The Important Part Because so many people had this question, and I finally talked to someone at Comcast who knew what Gigabit Pro actually is, here is how to get in touch with them:
It’s actually 3g up and down. It’s 10 gig Metro-E into a Juniper switch, with a 1g port and 10g port throttled to 2g.
I wish they had it in more places.
Their model is expensive because they don’t wire while neighborhoods. For me they had to run fiber a quarter mile, down my street, and trench under my driveway. I imagine all these one-offs make the per house investment quite high.
In another instance, I needed symmetric fiber at one of my businesses. I contacted Comcast, the business/commercial departments multiple times, got numerous peoples' names, left voice and email messages left and right and got nowhere after weeks. I ended up contacting a different ISP, signed up with them, and then on the install date, Comcast showed up to install the fiber modem (who were, of course, reselling it to the ISP I contracted with). I ended up paying the other ISP $1,700 per month.
The only real complaint I have is that Verizon still seems to be unable to deploy IPv6. They have promised it for years and never seem to get past the testing phase.
I still wish I had other options in my area (nh)
https://www.theverge.com/2021/1/21/22242635/fcc-ajit-pai-int...
Not "3mbps upload speed is all internet users need".
A fiber cut auto heals automatically via bgp. Absolutely no truth in this claim from verizon.
Meanwhile google, zoom, aws, and microsoft went down?
There isn't any shared datacenters here. There's at least 3 distinct datacenters.
So what's the truth? what actually went down?
I think this revealed that there is insufficient failover capacity.
I've even seen where playing it safe while planning to back out of a manual fix used during a weird outage (circuit went down in one direction or something, just needed to be manually shutdown to re-route everything) brought down the entire network when another outage happened before that evening, i.e. most redundancy plans protect against single failures, not all combination of two failures. They wanted to do all planned changes after business hours, like manually swinging traffic back to the primary hub site. They didn't check the weather on the other side of the country, and the power at their new super-duper data center went down that afternoon during a thunderstorm...
I wonder what percentage of fiber cuts don't cause noticeable outages. We'll never know, unless they let us root around in their ticket system.
As a customer it is fair to judge them on what you see, though. Some telcos screw up more than others. Best to use two.
(Typed over my phone, cuz my broadband is down!)
Too much capacity is centralized in certain providers, and if they go down there simply is no backup route that can handle the same traffic patterns.
Every time an outage like this happens without a following postmortem, I wonder if it was a surveillance oopsie.
https://en.wikipedia.org/wiki/Occam%27s_razor
Why is it that conspiracy theories seem to always require a more complex answer than the truth?
My point is, the Internet rerouting around damage doesn't explain the slowdown, mostly because it happened days after, not immediately, meaning the damaged infrastructure was not critical for the global internet.
The attacks unleashed a huge intelligence operation, and it's reasonable to expect a parallel intelligence operation online. If the latter caused a traffic bottleneck or not, we can't know because it was never addressed, it's speculation on my part.
One thing they always fall down is the idea that government bureaucracy is so quick and competent that they could do something like installing mass surveillance at a moments notice while the country is in chaos without it leaking due the next 20 years
It didn't have to be installed, it was already, although in a pre-Snowden era, governments doing Internet surveillance was the conspiracy theory. My speculation is that more international traffic got routed through the existing surveillance infrastructure, causing a bottleneck that slowly got better.