Redundant Array of Flaky Connections | Hacker News Reader

Redundant Array of Flaky Connections

sumologic.com

30 pointsjakozaur31 comments

I could use a RAFC right now for cell-based internet here in Russia. There are 4 providers for wireless internet (3 UMTS and one CDMA450) and none of the three UMTS work reliably. Haven't tested the last one yet.

Looks like the peplink guys (cited in the article) have just the thing for you:

http://www.peplink.com/max-mobile-router/

we have connection outages almost daily

Seriously? Is this considered acceptable for residential ISPs in California? Here I'd find one outage every month to be unacceptable, much less daily ones!

It depends how you define "outage". When we had an ADSL line it rarely went down, but regular ping tests along it often showed packet loss.

Yeah, for consumer level ISPs in California, sadly, this is a reality. I've used Comcast, AT&T U-Verse, sonic.net, Speakeasy, etc over the years, and they all had outages.

Amongst the better ones were Speakeasy and sonic.net. Comcast is amongst the worst. Especially here on Castro Street in downtown Mountain View, which is basically littered with tech startups, things are pretty bad.

Huh, that's odd. I live in Sunnyvale and have had very little trouble with Comcast connectivity for most of the last several years. There was one period of time when I was seeing frequent (> daily) connection loss, but that was cured by a cable modem upgrade.

Let me add some nuance here: We have the "business" offering from Comcast. It was the only offering that had 100MBit/s when we signed up. With the "business" offering, you are forced to use the "business" modem.

The modem is made by SMC. It is a full firewall/gateway. Unfortunately, the software on it is not very stable. For about 4-5 months, we worked with Comcast customer support to resolve this. They tried many different things to fix the issue, without success. My conclusion was that the software on the SMC modem just crashes once a day or so, and then hangs.

Eventually, I convinced them to let us use the "simple" residential Motorola modem. That has worked much better. It still crashes every 1-2 weeks, but that's almost acceptable.

One advantage not mentioned here is when you need to perform maintenance. Being able to take any single device offline without affecting the service means that you can perform software upgrades or hardware replacements with more flexibility (and often within working hours)

The biggest problem with this is the difference in ToS between consumer grade and business grade connections. If you try to use a consumer comcast cable connection for business you may run into things like bandwidth caps, filtered ports, etc.

You should definitely get business service if you're going to do this; it's not that much more expensive.

Yes, and that's actually what they did here, even though they label them "consumer grade" connections.

Looks great for client-side connections!

I wonder to what degree this is a solution for office-based servers. How would DNS failover be handled?

Even though we don't run any office-based servers at Sumo Logic, the recommended way is to make the Peplink a DNS server. It will then serve the IP of the currently active link...

They also just sent me this in an email:

"Our revolutionary SpeedFusion technology is now available. Adding a wealth of innovations that take Peplink/Pepwave gear to the next level, SpeedFusion provides Balance users a cost-effective, high-bandwidth, and reliable alternative to using expensive MPLS connections in their branch VPNs, while SpeedFusion 4G/3G Bonding gives MAX users the ultimate in speedy and dependable networking on the road. Blazing fast, secure, and easy to manage, SpeedFusion is part of Firmware 5.4, so download this new release and become part of the SpeedFusion revolution today."

http://www.peplink.com/speedfusion

Looks great and thanks for the tip.

There are several tricks you can use for DNS failover, but it's not generally regarded as a reliable option. DNS caching has all kinds of awkward gotchas. You could also host a reverse proxy somewhere reliable which just forwards things to you over the appropriate connection

If you need reliable hosting with multiple providers then you can get your own AS number and use BGP routing to provide failover. But that's more expensive by one or two orders of magnitude.

You could setup a cloud machine to have the fixed public IP that DNS points to and then have that connect to your office through both connections and do the balancing. A cheap 15$/month low-memory VPS from Amazon or Linode would probably be enough. I wonder if tinc[1] is easy to use to do the load-balancing.

[1] http://www.tinc-vpn.org/

We do basically the same thing at our office, though with IOS Event Manager foo sprinkled in. It works great.

1, "Set up multiple consumer grade internet connections."

2, Discover that all of them are just reselling the same service provider's product and they all go to the same cable in the same junction box under the street.

Even with totally independent enterprise grade 'N'-9s uptime guarantees we discovered they are in the same trench under the road outside our site. A lot of major data centers have discovered that it only takes a single idiot with a backhoe.

Perhaps checking trace route with mtr should give you the answer to 2.

Not with DSL.

At least in europe the last mile is always BT/Deutsche Telkom etc but the other ISPs put their adapters in their racks. So it shows a different ISP all the way to the host but they are on the same wire and the same PSU in the same cage.

Sure, but the vast majority of outages aren't related to a cut cable. I've done a ton of telecom work, and the thing I encountered most commonly was billing issues (ha!). Next up is equipment failure at either end of the line; bad card at the CO or bad modem on premisis. A distant third is outside issues like bad interconnects, and waaaaaaay out in forth is cut cables. It really doesn't happen all that often, even in big cities.

By going with DSL and cable, they may be in luck even on that side of things. In a lot of places, phone lines (DSL) were run underground way back when the cities were built. Cable came much later, so it is frequently run above ground on utility poles. This means that if someone backhoes your DSL cable, the coax on the pole for cable internet will still be fine.

Yes - most of our problems have been our provider = "we decided to process your office move from 6months ago AGAIN and have disconnected you!"

But unthinkingly deciding that having `N` systems means your risk of failure is `1/N` when you don't know how they are connected is a classic way of getting on the front page of the dailywtf.com !

Agreed there. I missed one cause that encompasses all of them: the customer has no idea what they actually have/don't have in terms of infrastructure, which negates their ability to effectively measure and manage redundancy.

I assume that's why they got both cable and DSL, although depending on how things work out they might still run in the same trench.

Trench? Wishful thinking in California. :) There's a big telephone pole behind our building, and both the cable and phone lines run from the building to the pole...

This setup is by no means good enough to run any mission critical services off of. It is, however, a big improvement over a single business line to keep a group of 30-40 people reliably connected to the internet.

It sounds like they still have several single points of failure. Most obviously the multi-WAN router - I doubt they paid double for a cold or warm spare unit.

Absolutely, the Peplink is a SPOF. They do support an HA mode with two Peplinks, but that hasn't been necessary for our uptime requirements yet.

It helps that the unit has been rock solid, too. Not a single crash or hickup.

We do have another (non-Peplink) router we can drop in within a few minutes, we've had to do that a few times when we were running another brand of router prior to the Peplink.

I've set up something similar with old hardware and pfSense.

It supports Multi-WAN and HA failover, and the price is nicer than the custom hardware units (as long as you can find a fast enough unit to run it on).

I love pfSense, it's a great platform! I used to run that at home for several years, and even in the early days at Sumo Logic.

The only achilles heel I've found is the fact that it runs on a PC, and PC hardware tends to have a lot more moving parts and break. You also need to take care to make configure the BIOS to turn on the PC after a power failure, etc.

pfSense is definitely a great option, but I've basically decided I'm too lazy and prefer the robustness of the Peplink. A smaller Peplink unit only runs $300-400, definitely comparable to an entry level PC...

LTE can help with this since it should be more diverse than cable and DSL.

We helped a client of ours setup something akin to this, to work around their (essentially unfixable) Internet situation.

We rolled our solution using OpenBSD/pf/carp/pfsync due to client expertise and comfort with BSD. It works well enough that outages on their lines have shown up only as nagios alerts, not actual outages, even through many years of service.

Reply on news.ycombinator.com