When Your Fiber ISP's 'Dumb' Device Screws Your TCP Sessions
neelc.org
neelc.org
Off topic, but CenturyLink Fiber still uses PPPoE and 6rd instead of native dual stack in many markets and are unwilling to upgrade to more modern configurations.
EDIT: I do not use Tor at all.
GSM solved provisioning 30 years ago with SIM cards, any reason why ONTs couldn't employ similar system?
I wonder how hard it would be to connect the Nokia 3fe4960ac SFP to a linux server and initiate a DHCP on their 802.11q vlan, or PPPoE, or whatever else they may use?
Yes, you can do just that. You can also use a commercial firewall with an sfp port.
SFP ports can be a little troublesome. There are a few standards (SFP/SFP+) and different link speeds.
The you need to know your ISP's specific info to configure your hardware. Bell uses pppoe on VLAN 35. You need your username/password for pppoe, which is not a given since the technician will usually do the initial config for you.
Another option is to use a media converter. They are very cheap and "dumb" devices, that simply converts the fiber to copper ethernet. You then connect using just about any router that supports vlans and pppoe.
If you also get bundled TV or Phone subscription on your fiber link, these will be on another VLAN and connection is much more obscure. Usually the logic is embedded in the all-in-one router they provide.
Knowing little about the GPON protocol, what does the ONT actually contain to authenticate to the network? With some quick web research, it seems like it's a serial number and/or a static password Would it be possible to replace the ONT with a well documented model that you have flashed with the appropriate identifiers?
You might have to figure out how to take the ISP's provisioning profile and make your own device use those parameters? Then again if the ISP didn't want dodgy devices on their TDM network they should remove the motivation by deploying non-broken gear in the first place.
Given that ONTs probably aren't subject to too much hardware security research, maybe it would be possible to hook up a debugger and NOP out the connection tracking hooks.
At the ISP before that we made our own CPE. The leads on that project really understood the Internet and managed to get reasonable latency, even over WiFi. But the incumbents still seem to not know about fq_codel, or how to put more than 4MB of RAM in their devices, and the users suffer as a result. This article reminded me of how mad it makes me, sorry for the rant. (I switched to a different industry where less lasers are involved.)
Hopefully some crazy person will launch an astronomically expensive fleet of satellites that solves these problems once and for all. All you need are a few engineers on the CPE team that give a damn, and you can fix the Internet for everyone in the world.
His ssh sessions were constantly timing out. It only happened when he left the SSH session to idle. It turns out his router was dropping the TCP sessions because it considered them dead. He got around it by implementing a "keep alive" packet, of sorts. Very interesting stuff. I don't really work at such a low level in the stack regularly, so it's quite fascinating to see the strange issues people encounter with these tools. Especially when ISP's meddle around with stable protocols.
Also reminds me of how some ISP DNS servers totally ignore TTL values from DNS records[0].
Given all the advances in technology, I don't think that's as bad an idea as it once was.
You can't just produce RST out of nowhere on the NAT expiry, because that connection may actually be active somewhere else. Consider a replicating pair of NATs - your connection gets moved from one to the other because (network reasons), but the previous one does not get a message about it because (network reasons). If it sent the RST packets it would actively kill live connections which it should not touch anymore.
Like, perhaps this problem is also affecting other kinds of usage, but the original article does not attempt to claim that, and purely from their example it would be generous to assume that literally dozens of individuals would need this feature and, well, it's not worth to make and test features (even if they're just a configuration option) in this case.
Which is all just one reason that, of the set of people running Tor relays on residential internet connections, I'd wager a solid 99% shouldn't be.
In general it's extremely unlikely unless you are engaging in "high risk behavior," but at the scale of an ISP there are enough users doing that kind of thing (Twitch streaming, etc) that it becomes an appreciable frustration for your network operations.
This sounds like the sort of thing with similar prevalence to things like running a Tor node. This might even be an example the other way, when your game server or what have you has thousands of peer connections and this thing breaks it by misinterprets that as a denial of service too.
If they say they have no idea what you're talking about, you get to tell them they're infected, so they actually fix it instead of typing their bank password into the infected box the next week because you automatically removed the "huh, internet's slow" that might have led them to investigate it otherwise.
I suspect the concern is not that ordinary users would be targets, but that ordinary users would be sources of ddoses (by being part of botnet)
Furthermore, you need some kind of ONT for fiber termination and it isn't clear that Centurylink uses a different ONT without this feature for business class customers.
Whoever thought a *stateful ONT* was a good idea should be shot out of a canon.
Just wait until the connection timers in the ONT don't match your firewall. Then you'll have real fun.
- your 95th percentile usage is now likely going to be substantially more than 4 Mbps
- your usage is likely to be much more constant (less bursty). This breaks statistical multiplexing amongst residential users. For reference, Netflix with HD video streams tends to burst to 25Mbps for a second and is then idle for 4-5 seconds.
- your usage is now exposing the ISP to DoS attacks and other interesting (read as expensive) problems caused by running a Tor node. This includes legal costs when dealing with investigations into malicious use of the network by nefarious people trying to hide illegal activities via Tor. Yes, your ISP has to bear the cost for legal issues that arise when its users engage in illegal activity over their internet connections.
- your Tor usage is likely to result in the IPs that are used by you to get added to various blacklists. This results in support costs for the ISP when your dynamic IP gets assigned to another user and causes problems for an unrelated.
If you really want to do this, colocate a Tor node in a data center. This kind of traffic is perfectly appropriate in commercial circumstances, and the price you pay will reflect the actual cost of the service being delivered. You're not going to cause nearly as much collateral damage with a dedicated internet connection as you will on a residential network.
Yes, Tor has its place, and if you're going to run a Tor node, think long and hard about the impact it will have before doing so. Many smaller ISPs are not at a scale where the company can afford to carry the costs needed to support traffic patterns that are generated by Tor. Small ISPs have to be very careful to balance the line between expanding to serve the needs of our customers and breaking even. Legal budgets only become a thing after an ISP has hundreds of thousands of dollars a month in revenue. Please, don't do something like this to a small ISP that's trying to help bridge the broadband divide. At the very least, run it by them before doing so.
Even regardless of the Tor issues, the problem OP is having is related to the quantity of TCP connections, not Tor itself. So the points are irrelevant. He could be connecting to arbitrary HTTP servers and run into the same problem.
I say more traffic going to true edge nodes is a good thing. The more vibrant the P2P ecosystem, the harder it becomes for ISPs to discriminate against communications not going to big tech, and the harder it is to monetize user surveillance. The more customers that view their connection as something for publishing and actively participating, rather than merely consuming, the better off we all are.
If you want to implement a bandwidth cap for your users, go right ahead. Just make sure to post it as prominently as your burstable speed. 10TB/month is 32 Mb/sec.
Total ballpark, because it depends plenty on your market, proximity to carrier resources, etc, gigabit symmetric DIA tends to be in the neighborhood of $1000-2000 per month. A lot of the variance comes from the fact that it will be delivered by conventional fiber, not PON, in order to avoid resource contention. So trenching is usually involved in the installation, but the price of that is usually amortized into your 3-year contract.
Sorry, that ship sailed long ago. Carriers have forever put restrictions on how their customers can use their internet connections, such as "no hosting servers" or even not getting a routable IP address. Traffic shaping is part of the deal too.
I think the only means we have to change the situation (in the face of a lack of competition) is to lobby for municipal internet. Or start a company.
I talk about the rationale here:
https://www.naut.ca/blog/2020/12/30/launching-a-new-service/
Anyways, wish all the best luck to you.
There's also this service which I have for Tor bridges: https://www.aceinnovative.com/internet-access/static-ip-vpn/
Ace's "Static IP VPN" gives you more IPv4 space and is uncapped, but forces a long-EOL Cisco router and is very slow (meaning 2010 speeds). I had this on-and-off, and pushed my main Internet traffic over it from 2013-2015, but it is less flexible.
I'd love to join Hoppy Network if it had a Seattle PoP, and if I could get an "uncapped" plan or at least a higher bandwidth version (even if it is super expensive). I don't demand it from you, though.
And at the same time I totally understand why you have limits (for real), you don't want people to abuse your service and slow down everything. Maybe I will join, who knows.
Can you explain? I thought TCP was implemented on top of UDP.
But UDP is basically IP + a port number and a checksum, while TCP is IP + a port number and includes a checksum as well, and a bunch of other stuff. So while TCP isn’t exactly implemented on top of UDP, it’s pretty close.
Something very much like TCP can be implemented on top of UDP, with whatever improvements or differences you might want to implement. (It’s just that both sides need to understand the custom TCP-like protocol.)
I've seen a few pieces of niche Free software that do such a thing to get around super restrictive firewalls.
But reading this, clearly one can have much worse.
Having any ISP competition at all is relatively new for the Seattle metro. It was essentially only Comcast for a decade. Now we have Centurylink in large parts of the city. I don’t think Google Fiber is widely available here.
Centurylink rates for 940/940 Mbps in Seattle are $65/mo, which isn’t as nice as $50, but not bad. Comcast charged like $80/mo (coax) for ~200/20 as recently as 2017.
Wave G is a crap service. Wave took CondoInternet and threw it down the drain. My "Gigabit" there was actually 10-20 Mbps most of the time. And where I had it, the building had wiring exclusivity to Comcast and Wave, so no Ziply Fiber or Atlas Networks. In fact, if I had to choose between Wave G or AT&T Fiber and it's forced router/802.1X, I'd pick AT&T any day.
From what I know, Wave G uses Layer3 Ethernet switches whereas CenturyLink uses GPON.
CenturyLink has a much better service which is sadly held back by the crappy ONT. But if you don't max out your TCP connection count and don't need IPv6 or can live with 6rd/HE.net tunnels, IMHO CenturyLink Fiber beats Wave G.
Google Fiber (actually Webpass) was real solid in my previous place, plus it has native IPv6 which CenturyLink lacks. CenturyLink 6rd sucks, and Hurricane Electric tunnels are getting blocked by sites now.
Most annoyingly, the FreeBSD mirrors are slow via CenturyLink 6rd so I have to route that via Tor since FreeBSD pkg lacks a IPv4-only flag.
CL does have slightly better IPv4 peering and throughput, but Google Fiber/Webpass is totally worth it for $5 more. Yes, Webpass is slightly "slower" on IPv4, due to microwave links, but the real IPv6 and support and even sticky IPs make up for me.
And it's not just tor relays that use a lot of TCP sessions. Pretty much all distributed protocols are going to hold open a lot of TCP connections. This is not a bad thing and it isn't a heavy resource usage. It's normal. What's abnormal are wireless telco style restrictions being applied in contexts where there is no justification for them.
Saying everyone who does more than use a browser should colocate at a DC is disconnected from reality.