"BGP at home": getting a DIA circuit installed at home
aaka.sh
aaka.sh
When you have an SLA, understand what it is: a financial arrangement whereby you can request a prorated refund for certain types of outages. It is not in any way a guarantee on the part of a provider that you'll experience even average uptime equaling or exceeding the SLA, just that they can pay out the fraction of customer requests for service credits they receive for the covered outages they have and still make money.
The reality for the type of service the author of this post purchased is that for any physical damage to the fiber plant, he will experience hours of outage while a splice crew locates and repairs the damage. Verizon might offer a 100% SLA, but they didn't engineer it to even five nines of availability. That would require redundant equipment and service entrances at his premises along with path diversity end-to-end.
1. Unless you are a large customer who accounts for an important amount of their bottom line, probably you have little financial leverage with the vendor.
2. The amount at stake in the SLA is not worth going to court for. It's unenforceable and in fact, the amount is usually meaningless.
Let's say you pay $10,000 per month for your 100% SLA dedicated circuit, and it goes down for an entire month. Let's say the vendor doesn't get around to paying you. Is it worth hiring a lawyer to collect $10K? Is it worth distracting you from your job emotionally and mentally, and consuming many hours of your time? Probably not.
Let's say your circuit is down for 3 hours. Let's say the SLA even pays you 3x what you pay for the service for any downtime (most I've seen just refund the money for that time). Let's see: ($10,000/month) / (720 hrs/month) = $13.89/hour. The SLA pays $41.67/hr of downtime, or $125 for your downtime. Is it even worth figuring out how to apply and filling out the form? No. You have much bigger issues in business, and if you don't ... well, then you have bigger issues.
3. The cost to the vendor is reputation: You tell your peers how much your service sucks, and word gets around. I've had techs take disinterested attitudes toward their poor uptime on our circuit; when I've called the account managers, they can have a very different response - they want me spreading their name around in a different way. That has nothing to do with the SLA.
> Verizon might offer a 100% SLA, but they didn't engineer it to even five nines of availability. That would require redundant equipment and service entrances at his premises along with path diversity end-to-end.
Agreed. There is no substitute for the physical reality of the circuit and service, which you should understand if you are buying it. Putting a shiny SLA on it will have no effect on the outcome.
Edit: Ah, “Our network. You acknowledge that We may change your Internet Protocol (IP) address from time to time without giving notice unless You have purchased a fixed IP address from us;”
One time I contacted the business contact with a bunch of addresses and that was useful but the round trip is so slow I was better off just querying.
Certainly in UK that could be also be quite a risky thing to do, as it could expose build strategies to competitors. That said, we expose an API to Better Internet Dashboard so they can do map based coverage exploration.
I am jealous of them save for the "your monthly ISP bill is similar to a car payment" aspect.
Such as over-subscription and having to contact the BBB to finally get a non-"Let me look at my book, ah yes! it's your modem" response. I finally was contacted by the Technical Operations Manager to affirm "Yes [name] is correct, bandwidth demand exceeds the capacity of the system in their area. We are working on a permanent solution to allocate more bandwidth"
That was 9 years ago, and I'm back again. I pay for the catchy "UP TO" 35 MBit/s upstream and can barely hold 2 MBit/s during peak and about ~25 outside peak.
Between fighting for bandwidth amidst everyone else going to the same base station, random assigned IP addresses that occasionally end up with accusations of pirating you had no partaking in, and storms messing with your signal quality, I would heavily advocate against any reliance on it for business related operations.
Starlink also does not offer static IP addresses.
They do, albeit I don't know the exact details as to pricing and everything. My company uses Starlink as a backup WAN connection at one of our sites, and it has a static IP.
> Although truly static IPs are not available, a reservation system retains the public IPv4 address and IPv6 prefix even when the system is off or rebooted.
You have a potentially stable address, but not a truly static one.
https://blog.apnic.net/2024/05/17/a-transport-protocols-view...
At work we deal with the sort of folks in this blog, where adding a link between a couple sites requires four or five months, multiple teams boring new fiber runs, etc.
In London UK we can get dedicated 1Gbps for ~450GBP/mo with 3k install fees.
Anything privacy related, running a TOR node etc. I get it.
I frequently see people default to AWS, without any consideration of any other options. If you're running beyond a couple of small EC2 instances, it's worth looking at other options such as colocation. 37signals wrote about their cloud exit and how much they saved.
While Egress pricing is a pain in the ass on AWS, that's usually a small fee on the customer side comparatively.
Ultimately I get the SLA, Direct access to cloud providers maximizing performance, i'm also able to host a few IP blocks which allow a couple internet facing machines.
The home we sold recently had it pretty good too though, ended up with 3 5gig AT&T lines (no redundancy obviously) for only $450 a month total. That was pretty darn rad, even if the SLA wasn't the same.
Benefit of working from the farm is that I can also snag some bw for personal use ;D
Getting an ISP to even _talk_ to me required quite a bit of sleuthing. And I was saying from the outset that I was ready to fully pay for the fiber run.
Apparently, ISPs in my locality actually divide the city into the service areas. How the heck this is legal, I don't understand.
Some tidbits from me: my ISP installed a big honking ADVA optical line terminal on my premises. Getting them to move it to their side and just provide me with an SFP connection is still my work-in-progress.
The support is also outsourced into India, and getting them to understand what you want over the phone is... painful. Fortunately, the web ticket system is good enough.
Lack of competition is more about the cost to establish service in an area, and the ROI on service in an area with competition. It costs a lot to pull wires past a lot of potential customers and if many of them won't sign up because they already have a good enough option, it doesn't make much sense to do it.
Cable and Telco compete because when cable was built, it was a completely different service, but they've both evolved to fill the same role.
This is why mandatory line sharing is important for competition, and it's in the telecom act of 1996. But the FCC first said it only applied to telecoms, and then said it doesn't apply in remote terminals because of lack of space, and then courts said it doesn't apply at all because telecom and not cable isn't fair.
If you get it, it can be great. Imagine your ISP calling you when you reboot your router.
It's possible that the termination equipment could be a bare SFP and not a big box, but the ISP wants to be able to monitor the status of your connection up to the termination equipment, because that is their responsibility to keep online, and they can't do that if it's just an off-the-shelf SFP. They probably wouldn't agree to do it and still have any SLA.
If there's a physical problem with the box (too big/loud) you can try negotiate for a different box but if you just want control over your network, sorry but that just isn't how it works. Your network starts at the demarcation, and you don't want to be responsible for speaking whatever protocols the ISP is using internally, either. Up to the demarcation, it's ISP internal network, and past that, it's your network, with a standard handover interface like Ethernet.
It used to be that you had to get a cable or DSL modem from your ISP. Now, cable and DSL networks are standardized enough that there's a good chance you can get a third-party one to work. It will be the same with PON in time.
I think it's fine to be required to use your ISP modem, since different ISPs have different physical layer networks. It starts to suck when they try to force you to use their router. I think in the EU it's actually a legal requirement that if they do, it has to support bridge mode (a.k.a. behave as only a modem).
It’s mostly the same with PON now. I’ve always had success with the FS.com PON SFPs, once I get the necessary information for the connection to program them with (the difficulty of which can vary from “ask the tech installing the connection nicely” to “take apart the ISP-provided CPE and solder wires to the debug console pads”).
There is, in my case. Their point of presence is about 800 meters away, and they have a simple switch that aggregates connections. Their optical terminal on my terminal (a switch-sized 1-U rack-mounted box with ANNOYING fans) does all the traffic shaping, authentication, etc.
> It's possible that the termination equipment could be a bare SFP and not a big box, but the ISP wants to be able to monitor the status of your connection up to the termination equipment, because that is their responsibility to keep online, and they can't do that if it's just an off-the-shelf SFP. They probably wouldn't agree to do it and still have any SLA.
Sure, and having equipment on my premises makes it much easier to debug the issue. But they actually monitor the BGP session state, not the optical path.
While fascinated with the network stack, I've only gotten as deep as reading Illustrated TCP/IP and pretending I understand tcpdump. I would love to rovel around in BGP and, um, all that jazz.
Any suggestions on how to get started? My vague understanding is that most people get apprenticed into this stuff through work. Are the relevant systems involved just too expensive and locked behind corporate walls to be amenable to autodidactism?
You’ll need some kind of business entity to have easier conversations with ARIN, which is the starting point for getting yourself an ASN.
Once you’ve got an ASN, you have an entity that can “own” IP blocks instead of just relying on other networks to handle that for you. Now, have a look at Neptune Networks’ offerings—they’ll rent you an instance for a reasonable monthly cost that they’ll allow transit to and from. Note that their smallest instance size doesn’t have enough RAM to store the global internet’s full BGP table; this will matter only once you know what that means.
That’ll definitely get you started, and you can learn a lot on the cheap before even looking into your local colocation options.
Stevens' book is also a stupendous resource for the down-and-dirty, so good work on starting there. Beyond that you need to start just building things.
Almost every virtualization suite allows you to create network resources (or at least it abstracts the low-level OS calls or commands required to do so). Set up two VMs. Make them talk. Break that link and learn how to repair it, using the tools that you've mentioned you are now using, tcpdump in particular. Figure out how ARP works at a low level, or NDP (neighbor discovery) if you're running IPv6. Learn how to subnet, too! Then work your way up the stack. Set up a VLAN interface, set a 802.1q tag on an interface, try to get two or more vlans to talk to each other, route between them. Break that. Set up a basic OSPF area. Set up a BGP adjacency between two private ASNs you have created. Redistribute routes among different protocols. Set up higher-level services like DNS. Set up a play anycast network on your local host. Play around with load balancers and web servers. Play, break, fix, repeat. That's pretty much what the 'professionals' do all day anyway. It all comes from practice. Software like BIRD, quagga, nginx, haproxy, ip/nftables, dnsmasq/powerdns, etc etc.
When you think you've exhausted the software side of the above tools and beyond and want to lay your hands on some actual hardware, look at picking up a 'white box' switch, a cast-off on eBay from the likes of Quanta/QCT, Edge-core or others. Don't spend more than a couple hundred bucks on this. Throw an ONIE network os on them (I suggest Sonic for open source, or if you want to pay, Cumulus) and start using 'real' hardware and play around with that. Learn the basics of sfp transceivers, fibre optics and the different mode types they come in, direct attach cables, port channels and the like. You can find super cheap transceiver hardware, fibre optic patch cables and all that at a discount vendor like fs.com, or from ebay as well. Learn how to interrogate the firmware on those, find out power transmission levels, error rates and other system info.
There's a sibling comment here suggesting you start out by setting up a LLC and going to ARIN and getting an autonomous system number. Please ignore that advice. You will be just wasting your time and your money and be distracted for no reason until you have the most basic of foundations. Use the abundance of resources you have to learn first. If you really feel like you want to take that next step, then be confident and do it!
Like a lot of things in this industry, the complexity can get fractal in nature the more you look at it. Don't let that overwhelm you. Take it one step at a time and don't be afraid to break shit, fixing it is how you learn best.
How does this work for FTTH? I know nothing about fibre optic networks. I had the impression that each subscriber has their own wavelength, or rather a range of wavelengths that captures their bandwidth, and that does not overlap with other subscribers.
Otherwise I have no idea how passive optical networks could even work.
The keyword to search for is GPON. It's multiplexed, each subscriber receives a few time slots in a shared wavelength. The transmissions from the subscribers don't collide because their time slots don't overlap.
The OLT (optical line terminal, head-end) will tell each ONU/ONT (optical network unit/terminal) how much airtime they can use to transmit - each ONT will take their turn in transmitting so as not to interrupt others. Part of this calculation is the distance the ONT is from the OLT - each ONT will be a different distance depending on the geographic location, which means each ONT will have a different latency. The ONU can request additional airtime if it has a large amound of data to transmit. The amount of airtime the OLT will allocate depends upon the CIR (committed information rate i.e. what will the ISP guarantee at a minimum) and the PIR (peak information rate - the maximum rate based on the subscribers service).
I don't work for TWC and have no their services. However friend of mine worked in past for a regional competitor of TWC around 2010 and explained the logic above.
And yeah the quality of customer service we’ve gotten from three different business providers has been exceptional. It’s crazy to have actual engineers you can call who know what’s going on. You get what you pay for.
I went down to the basement and saw a faded UUNET sticker on the demarc, but there was no circuit id on it. Some googling showed that through the years of corporate takeovers they were now owned by Verizon. So I called Verzion Business and explained the situation. The lady spent a hour on the phone with me, but we tracked down the circuit. The address listed on the circuit was a manhole up the street from our building. They dispatched a technician and we were back up and running in about an hour. They also put new circuit ID label on the demarc so we wouldn't go through that again.