Hacker News Reader

Comment by hedora | Hacker News Reader

With how many 9's of availability, and at what latency?

Frontier is advertising 3 nines average network-wide availability for their new fiber network. Three nines is completely unacceptable, but our neighborhood barely got one nine last year.

Based on their current behavior, there's zero chance of them providing the bandwidths they're advertising for the new build out, even when the network is completely up and completely idle. Currently, if we try to sign up, they say we can get <<1 mbps (bits) theoretical. However, the internet coverage maps claim they provide broadband service to our address, so they must be claiming > 25Mbps to the FCC.

View on HN

Why isn't 3 9's acceptable for residential home internet? That's less than 9 hours of downtime a year, 44 minutes/month.

Sure, I'd like 4 or even 5 9's, but I don't want to pay for it -- each additional 9 is exponentially more expensive to guarantee, and few people are willing to pay for that much reliability for their own network gear.

Actually averaging 3 9's isn't that bad. However, 3 9's looks much worse when you also miss the target.

The target for phones used to be significantly higher, but with cellphones home internet is generally less safety critical.

The average US customer in 2022 averaged 5.5 hours of power disruptions[0]. So somewhere between 3-4 9s.

[0] https://www.eia.gov/todayinenergy/detail.php?id=61303

4 nines is reasonable, 5 nines a good target. ISPs have become utilities, especially since covid.

The commercial internet services with 4 or 5 9 SLAs have multiple redundant lines to the building to guarantee this and they charge handsomely for it. Otherwise a downed telephone pole or something will automatically bring the service down for a day or more.

BTW I run a serious network at home (Proxmox cluster, all enterprise gear, virtualized router with HA, UPS power) and only target 4 nines on the LAN. System updates and things like server crashes making the router failover to another node cause around 30 minutes worth of network downtime a year. If a core or floor switch fails it'll automatically take me over an hour to swap in a spare and configure it. I don't bother trying for that fifth nine and I'm a very demanding user.

Now if one of you have a consistent 5 nine home system I'd be curious to learn more about it. I'm guessing that would practically be a datacenter setup with CARP, dual switches, etc. - or you're just really really lucky.

My electrical utility does not hit 4 nines, maybe not even 3. Water and sewer? Definitely.

5 nines is reasonable for electricity, if you live in an urban area with no extreme weather or other major natural hazards. When I was still living in Helsinki, blackouts were basically an once-in-a-decade thing, and they didn't last long.

Today in California, I'm getting 3 nines in an urban area. Those living a couple of blocks further towards the suburbs are not so lucky. And many people in the mountains don't get even 2 nines.

Hey, 3 nines ain’t too bad!

In 2023 our availability only had one nine. We had power ~96% of the year. And it wasn’t like it was a lot of hour long outages that added up or something. Longest was 8 days.

I am not in a dense urban area, but I am connected directly to a line connecting a couple of larger towns along a major highway, so we’re generally restored fairly quickly. During the 8 day outage, quite a few people in the major city nearby were actually still without power when mine was restored.

Maybe unsurprisingly, my internet has actually been vastly _more_ reliable.

That's sounds pretty terrible...

In my Eastern European country the SLA from the (state owned) power company is no more than 6 hours of down time per event, after which they need to pay compensation. Within highly populated areas it's shorter.

Admitidely the most remote place is less than 1.5 hours from the nearest reasonable sized town, so we aren't as remote as some parts of the US. The purpose of this was to force them to make their network more reliable by upgrading old equipment, burying overhead lines (lots of trees used to fall on lines during winter) and ensuring redundancy.

I don't have town sewer but being without water is very rare. Being without electricity for maybe the total of day a year? Not rare. (Though semi-rural albeit on a major road.)

ADDED: I don't. But people having backup generators in snow country is not a weird thing.

If they have to do maintenance on a water main, you'll easily get an hour of downtime, likely followed by a boil water advisory.

They replaced several hydrants near me a couple years ago and there were disruptions on a couple of days.

Dude, my DSL back in my second world country fared better. The US has enough money to have moderate standards

The DSL network piggybacked on the analog voice network which, at least in this country, was built to a higher standard. But I don't want to go back to the days of the 1990's where I was paying $40/month (around $100/month in today's dollars) for an analog phone line.

It was reliable, so reliable that "as reliable as dial tone" was something to aspire to, but also costly and limited in capability. Even if I could get DSL today, my fastest speed would top out around 12mbit/sec due to my distance to the CO, which doesn't even meet the current definition of broadband.

> Three nines is completely unacceptable

Less than 9 hrs of downtime per year is unacceptable for consumer internet?

Traditionally, it was unacceptable for landline service, since it's safety critical. (You need to be able to call 911.)

I poked around to try to figure out what the current requirements are, and it looks like they've mostly been scrapped. "Central office backup power" is required, which matches my experience attempting to get terrestrial internet or cell service during power outages.

If we limit it to issues that the ISP owns, then I'd say yes.

What does the ISP have that can cause 9 hours worth of outages throughout the year? The only thing I could think of is the ISP being targeted by a massive DDOS attack.

What does the ISP have that can cause 9 hours worth of outages throughout the year

They have the real world -- ice storms, trees blown down by wind storms, power outages, backhoe outages, etc.

A tree took out internet in my neighborhood for 2 days, others had longer outages since it was a big storm and it took a while to repair.

It's easy to have a single outage over 9 hours if they wait 8 hours to send somebody out.

lol, well yes I guess :D.

I guess I'm really just spoiled. In the last 10 years I think I've only really experienced like 1 or 2 outages caused by my current ISP and they were resolved within the hour. My assumption is that like most people, all the ISP equipment is buried so the only things that can really go wrong are either at the colo or the home (like a router that dies). Having a line cut is simply a hyper rare event.

On the other hand, my overhead powerlines have caused most of my outages. An idiot crashing into the telephone poll has made my power less reliable than my internet.

Having a line in the ground won't save you when there are wild backhoes in your biome.

Not every year, but Rogers in Canada managed to black itself out across digital TV, corp circuits, home internet, mobile, 3rd party internet providers that used their lines, their own employees, and cellular for the good part of a day (from like 5AM until midnight) nationwide.

Their CTO was on holiday and was unreachable because their phone went into SOS mode and they just assumed it was them.

https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...

In 2021, their mobile network went down for like 9 hours: https://www.thestar.com/news/gta/rogers-identifies-root-caus...

The Nashville AT&T explosion didn't take out the data lines, but took out the electric service. Then the gas company cutoff the gas lines so their backup generators couldn't run (wiki says it was due to fire/water damage, so maybe I have that part wrong). The facility didn't have roll-up generator connections, so they had to figure out how to wire them in at a big time cost.

https://en.wikipedia.org/wiki/2020_Nashville_bombing

In 2003, the "Slammer" worm slowed down ISP's core routers (including Rogers' to the point of not working anymore) with the quantity of small packets.

https://en.wikipedia.org/wiki/SQL_Slammer

An ice storm taking down a utility pole will blow that out of the water for several years.

I live in semi-rural Canada and get better service than that, but I would be fine with more downtime.

OTOH, I frequently have scheduled power outage times that exceed that metric so it sort of doesn't matter, and that's ignoring unscheduled power outages (a necessary part of life in a temperate rain forest with above ground power). I've taken to keeping a small generator handy to power the modem.

Our Internet + phone service go down if there's a power outage, which implies the cell network also goes down in practice. The tower might be up and on backup power, but it either has nothing to talk to or it's completely overwhelmed.

It is when you work from home.

I would be annoyed at losing a day or so of work, if the outage was all at once - but nine hours a year is tolerable. I wish my ISP was that reliable. Hell, I wish my power company was that reliable. Where I live it's common to have a number of outages longer than eight hours every year. Last year a moderate storm caused power outages for a couple of days in August affecting thousands.

Generously, if you work 2000 hrs/year, that's less than 25% of a year's total hours, so it's only 2 hrs of work downtime on average. If you need better connectivity than that, you should pay for a higher SLA or make sure you can tether a cell connection or something.

It is? 44 minutes outage per month is critical?

I doubt that's true for most jobs. But yes, there are jobs where it is, and some of us (e.g. me ;) are just finicky about network connectivity. In which case I highly recommend having a fallback connection, with a different modality than the main one. For most of us, that means LTE. Tethering is fine if you can live with <5 min outages, but sure, you can also do automated failover.

But it's not a level of service that I think is reasonable to expect from a single ISP connection. A single construction incident will burn the annual SLO budget, and there's nothing the provider can do about that.

(And of course, it'd still be nice if telcos ran their backends competently. Looking at you, AT&T)

Often you can use cell phone or a coffee shop if it's say 2 days/year of internet downtime.

If you need it for an emergency call, you need higher reliability, but you can prioritize, and I don't know how common landlines are any more.

Yes, given that that's also the phone line for many people and therefore is an emergency service...

Depends on how the 9 hrs of downtime is distributed. It could be 6480 5-second outages (packet loss). There are 3650 10-hour workdays in a year. If those 6480 5-second dropouts occur during waking hours, you're looking at 2 5-second dropouts every hour, which is going to suck.

I had a bad connection because of bad coax cable, and it's infuriating because each dropout causes long delays while TCP recovers.

Imagine all your streaming video or calls froze for a few seconds every hour for days on end. 9 hours in one continuous outage is fine, constant short outages are infuriating.

I'd be fine with 99.9% availability to my house. That means it's down for at most 9 hours per year. That's probably on par with how much my electricity goes out during an average winter.

But what they mean by "99.9% network-wide availability" and what I mean by "99.9% availability to my house" are very, very different things.

That’s crazy. If my electric was out for that long consistently I’d lose my shit. That’s totally unacceptable.

Don't live in the PNW then. Soil conditions make undergrounding utilities very expensive, and there's lots of trees near power lines. My first winter we had a 60 hour stretch with no utility power. The CenturyLink DSL remote terminal has no batteries, so if I lose utility power, I also usually lose internet (but I've had two power outages due to a downed line between the RT and my house so far). Most of the cell towers have batteries, but only 2-4 hours worth and then I've got no comms. My house has a backup generator, but it doesn't serve the well, so if I want water, I've got to drag a portable generator out there. A lot of my neighbors have generators, but not all of them.

Oh, and then there was the time a building contractor accidentally took out the underwater cable connecting CenturyLink to Seattle... turns out they don't have multiple paths for that, that was about 48 hours of no internet for me there too. I remember some cell towers didn't work, but I can't remember if it was by tower or by carrier.

In an urban area it would be unacceptable. It is not atypical for a rural area.

I think there is a gap in awareness of this depending on where people live. My rural hometown has frequent power outages, so many people have backup generators to at least keep the furnace running.

The town got popular with city people going remote during the pandemic, many of whom bought houses. I was visiting my parents a couple of years ago when the power went out for 18 hours or so during a deep cold snap. I was shocked that none of the new neighbors–who had recently completed huge costly renovation projects–had bothered to install an automatic generator.

Yeah. I don't know the exact numbers but having power out for 9 hours a year is not a freak event where I live in semi-rural New England and my impression is that it isn't for my colleagues in North Carolina either. Branches and trees come down on powerlines. I had a big tree come down in a windstorm a few months back. Lucky it didn't hit anything of mine or my neighbor or might have been a couple days.

I guess we can count you out the apocalypse then?

Its hard to prevent all power lines from going down in a storm, and as a bonus it would take down your internet as well, messing with its 9s

I was in Kazakhstan for some time. No electricity, clean/cold/hot water, heating for days at a time. In winter and in summer.

It could also mean nine one hour outages during working hours or one out of 1000 web browser requests fail on average.

However, I suspect that the actual availability is extremely skewed, given the one nine they delivered to our area.

> It could also mean nine one hour outages during working hours or one out of 1000 web browser requests fail on average.

Totally fine with both of these, even as someone who works from home.

> However, I suspect that the actual availability is extremely skewed, given the one nine they delivered to our area.

And this is what I'd find unacceptable.

You can and should challenge the map then. The FCC actually has a means to do that and does respond. Ideally get an email follow up stating speeds, frontier has a habit of refusing to create a paper trail.

https://help.bdc.fcc.gov/hc/en-us/articles/10476040597787-Ho...

I did this about 18 months ago, and surprisingly, Spectrum came out and actually put a new line up my road. I'd asked them off and on for years to get broadband. Last time I'd been quoted 12k to cross the little creek they stopped at and service the dozen or so houses on my side. But for whatever reason, this time they did it for free (well, normal hookup costs) after I challenged them on the FCC site.

Glad to hear that. I've been fighting to get Frontier to service my neighborhood and the best they could tell me is that there is some hidden ticket and they have no idea what the real status is.

Frontier is labeled as a fiber provider for my address on the New York State PSC Broadband map and I made sure to provide some feedback to them about that claim.

Would love to have even just a single alternative option besides satellite internet. I'm only 10 miles outside of the downtown area.

The FCC is working on an internet service notice format “modeled after the FDA nutrition labels” [1].

[1] https://www.fcc.gov/broadbandlabels

Ah, so one with huge amounts of input from lobbyists.

It is ridiculous that nutrition labels are not forced to standardize on something like x/100 grams (even Freedom units would be acceptable). Plus a x/package. None of this 3/8 serving size nonsense.

I don't see three 9's as being unreasonable for home internet at all, getting more reliability than that is non-trivial and most people have redundancy in the form of internet on their phones or can buy a second line.

3 9s is completely fine. Internet is not subject to the 911 uptime requirements.

3 9s is essentially an outage or two from a major storm. That’s fine for residential internet.

> 3 9s is completely fine. Internet is not subject to the 911 uptime requirements.

It's all going over the same wire. Why not put in the extra fraction of effort?

> It's all going over the same wire

Yes and no. Even if it's going over the same wire, there are different data flows. Voice uses predictable amounts of bandwidth per phone number.

Minimum speed still requires almost all the equipment to be running.

Yes and no.

* Last mile has a downed line, yes that will result in a local outage

* Backhaul gets taken out. Re-route traffic. Drop/throttle non-SLA traffic

If you're able to reroute then you should almost always be able to keep Internet service running, just slower. Not many connections are so weak that they can only carry voice.

> our neighborhood barely got one nine last year

Internet it available less than 10% of the year where you live ?

One nine should be 90% availability I would imagine.

Yeah that makes sense, I feel stupid now.

It was out for about a month total for some of our neighbors. (Repeated outages, not just one.)

Reply on news.ycombinator.com