How a Potato Saved Shopify's Internet

The problem was that all cabling had to be done underground at our place, and phone lines were the only thing we had in abundance. Coax, fibre, anything else basically meant busting up a path to the road in a major urban area. Plus it was the middle of winter which makes road work even harder.

Not every internet provider realised this immediately. We started down the path to better internet a lot of times with companies that said "no problem, we can get a wire to your building just fine" -- and then backed down weeks later when they saw our situation.

After one or two of these delays, we applied the shotgun approach, asking every single internet provider to give us an estimate all at once, to make the delays concurrent rather than successive. Every last one came back negative, or just stopped returning our calls(!). So Potato was my response to our complete lack of options.

The multiple IPs thing was an issue for some sites, but it turned out to work fine in general. If there was a site that really had a problem with it, it was also typically a low volume site, and didn't need heavy load balancing. So we had one of our proxies set up to not be balanced and just pick a single working link, and used that for those sites.

The problem is that DSL around here is PPP over Ethernet, rather than just plain Ethernet. So there's no MII, no ARP, no regular ethernet link bonding. It's hard to escape PPPoE for DSL around here due to phone company politics.

The type of bonding I talked about in the article was all PPP link bonding, which apparently still has a lot of issues to work out in Linux. Our two-line bond was pretty unreliable, and our attempts at a three-line bond ended up with a lot of return packets going nowhere. Didn't matter which lines we picked, either.