25 Gbit/s at home, part 1 | Hacker News Reader

25 Gbit/s at home, part 1

boredengineer.medium.com

131 pointspavel_odintsov149 comments

Recently moved up to 3gbps bidirectional and had a hell of a time getting 3gbps even with 10gbps network card directly into the 10gbps port on the ISP-provided modem. In addition to this, there are very few services that will actually feed you data at 3gbps.

Steam - ~280MB/s (2.2gbps)

Battle.net launcher - ~140MB/s (1.1gbps)

25GB Torrent with 1000+ seeders - ~70MB/s (~560mbps)

2GB iso from github - ~80MB/s (~650mbps)

fast.com - 1.4gbps

speedtest.net - 2.7gbps (using ISPs endpoint 2ms away)

Using a download manager like IDM or jDownloader will help for http downloads, but most hosts will limit your speed even with 16 connections open. I've managed to see 2gbps moving data to/from servers (scp) with softether configured to use 16 connections. The reality is with a single connection (majority of ssl transfers) you'll be limited by the sending side in almost all cases.

Overall it seems that while you can get connected and run an iperf to your ISP or multi-connection speedtest to a server hosted by your ISP or peered with your ISP, you'll be pretty much limited to <1gbps speeds regardless of your home network throughput.

Knowing this I would have simply went multi-gig (2.5g) for all in-home networking and saved a good chunk of change on networking equipment.

280MB/s steam? On a good day I get 25MB/s (Vodafone 400/25) on any German mirror

Speeds seem to vary based on the game and based on releases, interestingly. Some larger new-releases (Baldur's gate) were only hitting <100MB/s. I imagine it comes down to how loaded up the local steam POP is and maybe even the specific server hosting that game's files.

Aside from that release, almost everything that is a good size (>10GB) has gotten north of 100MB/s pretty regularly.

Your max theoretical would be 50MB/s on that connection, so 25MB is not terrible. I think there is some notable overhead on steam (check summing or encryption) as with my 1gbps connection previously I would never get past 100MB/s (800mbps).

I regularly max out at 70MB/s for other services, so the connection is actually pretty good IMHO

I get about the same; full link speed at 2Gbit. Steam will absolutely chew CPU to support this

I see a notable difference on a 7950X3D simply by messing around with the power/performance governors.

cable is shared media

Indeed, but no other service I use suffers from this to this extend.

And here I am with 300Mbps (Charter/Spectrum) that tanks to 50Mbps after 5:00 PM for the "low" price of $99.99/month...

Yeah, that is also what I found. 'Just' 1gbs fiber, but downloads are often bottlenecked by the server, especially things like GitHub.

the one nice thing is copying big files to and from my work computer at 100 MB/s, but my work network is mostly 1 Gbps anyway so it wouldn't get any faster if I had faster internet...

You sure it is the server and not routing?

Indeed. My cheap router, when bufferbloat mitigation is turned on (codel), maxes out its CPU at 650Mbps.

Its not just your router, also intermediate hops. I saw this same thing back when 1gbps was first being rolled out 20 years ago. My downloads would hard lock at 100mbps from a lot of sites while other locations would pull at the full gbps speed. Turns out some router in the middle would max out at a much lower limit. OPs test is showing the same thing. FWIW, Steam and Battle.net use P2P to distribute data so you get multiple streams using different routes. But GitHub, as example, has a single route between their datacenter and you and goes through a bunch of routing shenanigans once you hit their POP.

Get ready for a whirlwind of delayed network improvements from your favorite transit provider for the next 3-5 years just as we saw in the early 2000s!

Nope. There are plenty of servers that do push 1 Gbit, my provider has very good peering, and really close to several CDNs.

On 5/5 I've found Usenet obliterates torrenting in the throughput department.

Yeah but you pay for UseNet access separately. Some usenet servers charge extra for access over 100mbps or 1gbps.

You do need to pay but I would stay away from any provider still charging more based on how much bandwidth or transfer you're using, they are likely just extremely outdated or plain overpriced not actually better. Frugal Usenet for $50/y is where I'd point anyone but there are tons of good alternatives too so that's not the only place people should check out by any means. I've pulled about 50 TB in the last 2 months (NAS upgrade) and most files come in around 4 gbps out of 5.

Is there a modern guide to UseNet that you'd recommend?

I followed the guide here https://www.reddit.com/r/usenet/wiki/index/. I ended up doing Frugal for the provider, nzbgeek for the indexer, and nzbget for the client. Mapping to torrent terms the indexer is the torrent search site, the client is the client, and the provider is (effectively, in this mapping) who you pay to leech from. Depending how deep you get into it it may be worthwhile to look at a second provider on a different backbone (for greater availability) and/or another indexer but I'd start with just that and see how it works for you then add if needed.

Config wise signing up a provider is like buying anything else online. Setting up the client isn't much more difficult than setting up a torrent client (select default folders, set limits, etc) beyond you need to manually enter the info to connect to the providers.

Which is why you break the task into multiple threads.

Any sane Usenet client will do this by default but limitations from providers are per account not per connection.

Steam downloads are always very CPU intensive for me, it seems to need about 1 core per 100 megabit/s. I'm guessing they're using somewhat slow compression, perhaps LZMA. Likewise, game file validation is CPU limited (it only uses a single thread) at around 500 MB/s for me. Probably an outdated algorithm choice as well.

Unless you are say say pro competitive gamer (lag) or doing sone fringe science, anything above 1gbs real speed is useless these days.

I have it, or slightly below, with pings around 7ms, and... as a regular user with some modest 10tb movies and games collection, 99% of the cases 100mbit is more than fine, and the rest covers rest.

No place for anxious ocd types, but low ping and overall reliability of the connection are way more important to me, even for fetching 70gb+ games. Not only due to wfh, but also for that.

Or if you want to download a 4k torrent on the fly. A 40gb file will take only 2 to 5 minutes to download. Almost instantly.

Multiplayer games usually care more about latency than throughput. Modern FPSs can use as little as 250-500 kbps of bandwidth, but they'll play like crap if your RTT to the game server is >100ms.

True, but in my case, reliability and even price (per month) of going to 10 Gbps vs. 25 Gbps is the same.

And what I see as a use case - is when multiple people in my household would do heavy downloads simultaneously. Also, I hope I will get better-downloading speed (Steam, GoG, etc.) as with modern games, it becomes more critical, and I'll be keener to play and remove a game if I want to put in on pause, compared to keeping 3-4 100+ GB games installed at the same time.

I get the same impression using PS+ and trying to stream some AAA games. Even though I have a 1gbps connection, I'll get timed out from my VM running Deus Ex: Mankind Divided just trying to load a new map sector.

Not sure what you see, but I always multiply transfers times 10 to estimate bandwidth, since there is some overhead in the TCP/IP protocol. So 280MB/s I would estimate being about 2.8gbps...

To convert MB/s to gb/s (MegaBytes per second to Gigabits per second) simply multiply by 8 (8 bits per byte). So 280MB/s is 2.24gbps (moving the decimal place).

> but I always multiply transfers times 10 to estimate bandwidth, since there is some overhead in the TCP/IP protocol

20%ish seems high though, especially for file transfers which are probably almost all full packets near 1500 MTU cross WAN.

There is another variable as well. Most network equipment I've worked with reported throughput with incorrectly labeled kibi/mibi/gibi, and the ISP uses kilo/mega/giga.

The former system is multiples of 1024, the latter the mathematical and legal definition of x1000

The X10 rule of thumb is probably pretty spot on at the end

What godforsaken gear reports in binary prefixes? That's just... horrendous.

I think ethernet (1500mtu standard) has about 30 bytes of packet routing information per (max) 1500 bytes of data. That works out to 2% minimum. On smaller data packets (less than 1.5kb) it would be a much higher percentage. TLS on top of that, once you're out of the handshake, is about 40 bytes. those 40 bytes would be inside that 1500 window. Assuming 1460 bytes of data (best case) with 40 bytes of TLS and 30 bytes of ethernet overhead - that's up to ~5%.

There maybe further overhead when that packet gets to your router/modem and out to your ISP, and their switches, consider that there's likely another wrapped header on there accounting for 30 more bytes and possibly causing a packet to get split into two smaller ones. I think 10% would be the highest "data rate" vs "port rate" overhead you'd see, without involving a VPN or complex network.

I am not sure if the original poster is talking about downloading over https and, whether he is mentioning the data rate or the file size. For instance you download a 1gbyte file in 4 seconds - your effective data rate in terms of what you want to see on your disk is 250Mbytes/second ; but if you measure the actual data rate it would be more due to both TCP/IP and, application layer overhead.

There is a more concrete example - in the article I've showed iperf3 output, which gave 23.5 Gbit/s - that is actual data. Raw speed was ~24.9-25. Everything else is overhead in that case.

The bigger benefit of all of that bandwidth is going to be multiple users. If you work from home and have a big family, or just live with a few roommates you're going to be seeing huge quality of service improvements from one person not being able to tank the whole connection.

Not with my 800k DSL you won't.

Yes, this is why I went with the medium 10gbps package from Init7, with only a 1gbps internal network. Each port is 100% independent bandwidth-wise, and maybe I'll eventually add a 2.5gbps adapter on a suitable host.

I just upgraded to 10GbE service from Ziply Fiber in the Seattle area:

Speedtest.net - ~9.3Gbps symmetric

iperf to Denver: ~9.3Gbps symmetric (~8 threads)

iperf to Minneapolis: ~9.3Gbps symmetric

AWS S3 Download with small files: 1.3Gbps

Usenet download: 7-9.5Gbps

Mounting my ZFS pool to an AWS instance via SMBv3 in us-west-2: 3Gbps (not clear what the limit was here)

GitHub (400MB): 1 thread 400Mbps, 16 threads: 3.3Gbps

Fedora ISO from local mirror: 1 thread: 1.4Gbps, 16 threads: 4Gbps

Will try to test Steam and such.

Well, now you're just making the rest of us jealous.

AWS S3 Download with small files: 1.3Gbps

That's $53/h in data transfer costs from us-west-2... (They are so absurd.)

>Mounting my ZFS pool to an AWS instance via SMBv3 in us-west-2: 3Gbps (not clear what the limit was here)

this one sounds interesting and definitely a writeup I would read about.

I had some media files I wanted to verify the integrity of -- I was using tdarr.io in a podman container on my local server. I then spun up an m7a.12xl spot instance, mounted my local storage to it via SMBv3. I ran tdarr on that instance, and connected it as a worker node to my home server instance. tdarr ran ffmpeg to decode every video frame to /dev/null, and reported any errors. It was able to process a few thousand FPS or about 2-3Gbps. I found some corrupted files, and was able to restore them from another backup.

There was no practical reason to do this, it cost about $100 to double my processing speed.

These kinds of speeds are available in Switzerland because their laws state that any fiber laid down must be usable by competitors, so that ISPs must compete on service alone. I wish we had that in other countries as well.

But then who pays for laying down the fiber? What company would want to pay that cost when others can use it?

The company pays but then it can also use other companies' fiber, so sometimes it's worth it to do so in order to show that they have coverage in a certain area where there might not be fiber (that they then lay down), and also that they cover everywhere else.

It's quite a clever system that uses market greed to incentivize building. It's the same system as railroads, imagine if railroads could not share tracks with each other.

https://en.wikipedia.org/wiki/Arrangements_between_railroads...

Grants from the government.

Some places also vote to have the city or town connect with Fiber which then puts tax money aside for it.

In many places it is the power companies that run fiber and they are mostly partially or fully state owned. The lagest ISP also has the state as the majority share holder.

The government doesn't pay for anything.

It's whether you want to pay through it via tax dollars or directly.

By government I mean the tax payer. The government made the decision to allocate tax payer money for such things.

As it stimulates the economy similar to how laying new train track does.

In my case it was the city. I'm not sure if the providers pay for access or is it free (paid for by taxes).

Here in New Zealand the tax payer contracted a lines company to do it. ISPs compete on service but we don’t get 25 - maybe 4 or 8Gbps.

Usually there's a separation between transport (physical) and the service (billing and so on). Not the same companies. At least, in Italy there is.

It's the same for power, one or two companies lay the cable, myriad others sell the service.

We had it in the US but the Courts overruled Congress & demanded the FCC carve out an exception of the required Unbundled Local Element/Local Loop Unbundling requirements of the Telecommunications Act of 1996. By 2001 it was clear we weren't getting unbundled network elements after all. https://docs.fcc.gov/public/attachments/FCC-03-36A1.pdf

It was & is one of the most stunning activist court decisions to have hit the US ever. Big companies sued, saying fiber was expensive & that they should have sole control over next gen telecommunications technology & the courts ignored the legislative act & handed incumbent powers total control. Revilesome; one of the biggest constraints in US infrastructure in general.

The FCC has been busy 5hese past couple years unwinding remaining requirements for dark fiber & other remaining protections. I somewhat care but the obliteration of last mile protections that enabled competitive carriers to exist at all makes the overall market so unavailable & locked up that it's less relevant.

This is the way! And not just fiber, but infrastructure in general.

It’s basically combining the short-sighted greed of corporations with the long term potential energy of infra. The combo is incredibly powerful.

Governments typically suck at ops, pricing etc, but digging and drawing lines in the sand they can be pretty good at. Corporations without competition always suck, it’s like having an athlete without a sport. Even if they start out fine, they deteriorate on the couch into an agitated and lazy asshole, monopolizing the remote.

In many parts of Europe, this model is working so well that there are 2-3 competitors that are all good and cheap. Unlike (here) in the US where there is only one, depending on your address, which sucks and is incredibly overpriced.

> These kinds of speeds are available in Switzerland because their laws state that any fiber laid down must be usable by competitors, so that ISPs must compete on service alone. I wish we had that in other countries as well.

In Italy, nobody would lay fiber to stunt the competition.

I think it is same in Germany too. But I have a copper cable.

What is the state of the art in leveraging your home network to have a presence (website or otherwise) on the internet?

Fiber just became available in our neighborhood recently and I'm about to have fewer people sharing our network, so the idea of being able to host my own files is becoming attractive again.

We are about to be due for this pendulum (centralized vs distributed) to swing back again and I like to be prepared.

As long as you have a static IP address, this is trivial and typically very reliable. Throw an ATS battery into the mix and the internet will be super reliant, even when the power goes down.

Am doing this at home with a fibre line and it's been super reliable.

Dynamic dns is a thing?

My ISP uses dynamic dns, but it never changes unless my modem is down for several minutes. My router has a script that updates a dns entry if it changes, but realistically this rarely happens.

The choice for my fibre ISP is between static IPv4 or carrier-grade NAT. Both options get IPv6.

Cloudflare tunnels are easy to set up. Otherwise, serving directly can be an option if you have a public static IP from your ISP or a public dynamic ISP and are okay with doing dynamic DNS.

IMHO the key is to have a good router/firewall that gives you the flexibility you need—ideally a custom box like OP. My software preference is opnsense, but there are plenty of other good options.

I'm hosting my blog on machines running at home accessible via a Cloudflare tunnel. It works very well in my experience. And it's nice not having ports open to the internet or having to mess with dynamic dns.

You can run a website off a cellular connection, you don't need fiber for that. The pendulum has never left decentralized from a tech standpoint, folks are just too lazy.

Home computer ownership is tanking, millennials were probably the last "power user" generation, and there's just no major interest in self hosting with the convenience of cloud. Even among tech professionals like HN.

r/selfhosted would like to have word with you. That sub is increasing steadily.

https://www.reddit.com/r/selfhosted

I'm a member, I even built a self hosted home organizer (https://homechart.app). It's a niche community with a very high cost of entry (time/money/skill).

Regarding the money aspects of the high cost of entry - you can source cheap hardware from secondary markets. I just helped a friend setup a cheap selfhosting/homelab for his home. He bought couple of HP EliteDesk 800 G4 minis for $99/each with i5-8500T CPUs. Each came with 16gb ram. Both units had NVMe drives. So we just had to add couple of 2.5 SSDs 2tb ($50/each) for media storage. Running Ubuntu server with CasaOS ontop with handful of containers for media consumption (e.g Jellyfin) on one unit. The i5-8500T is new enough and supports Intel quicksync. So no GPU needed for transcoding. The second unit is running cloudflare tunnel daemon for things like his MicroBin(like pastebin), Wireguard VPN and NextCloud for external access. He still has plenty of room to add more applications to both servers.

Where'd you find those, ebay? That sounds like a great deal.

Not the GP, but eBay is great for off-lease or just plain old computer equipment.

Amazon Renewed. But Ebay has them as well. Often a bit cheaper on ebay. If you spend a bit more time searching.

I think the pendulum DID swing, as The Cloud became more popular and connections became radically asymmetric with very slow upload.

10 Mbps upload is plenty fast for hosting things like websites and web apps.

10Mbps is barely enough to send the ACKs on a 250Mbps download. Maybe it's fine as a dedicated connection, but you might not want to be hosting anything on your home connection.

Not really. For my daughter's birthday, I put together an album of around 100 photos to share with family. That came out to 386 MB at default quality settings in digikam, which would take 5 minutes to download at 10 Mb/s. Even for smaller things where maybe I share 10-15 photos from a day out, that'd take ~30 seconds to load. I can drop the quality just for sharing on the web, but I'd rather be able to just share it.

Likewise with backups, each photo is ~40 MB for RAW files, so 10 Mb/s can take all day to backup a couple hundred photos.

That's without even getting into video. Off the camera, video files are ~200 Mb/s. Backups of the raw footage take forever, and sharing at the original quality is a non-starter.

> 10 Mbps upload is plenty fast for hosting things like websites and web apps.

I made no guarantees for file sharing or backups...

I would consider things like home movies and photo albums to be one of the main use-cases for a personal website (photo albums are pretty much what everyone in my social network uses Facebook for today). If you restrict to text websites, we could all get by with < 1Mb/s internet connections.

Backups are something else sure, but for personal hosting, the world is easier if you can just plop photos into a directory with maybe a simple html (or in my case xml) file and not deal with things like generating thumbnails or different quality settings.

The story with the Cloud is that everything is distributed. The facts on the ground are that everything is more consolidated than it has ever been, mostly into 3 companies, and dozens of datacenters each.

I don't think the Internet has been this small in that respect since the early 90's.

As someone with a full homelab with all the fixins, I don't self-host external-facing services without a VPN (meaning, no ports open into my network and thus no public web sites etc). This isn't because I don't know how, or that I'm too lazy, but rather an acknowledgement that it's a huge security risk and it isn't necessary.

My public facing stuff lives on cheap cloud services and is kept well away from my home network.

> Just then, a bandwidth load as heavy as a pregnant elephant sits down on Manfred's head and sends clumps of humongous pixilation flickering across his sensorium: Around the world, five million or so geeks are bouncing on his home site, a digital flash crowd alerted by a posting from the other side of the bar. Manfred winces. "I really came here to talk about the economic exploitation of space travel, but I've just been slashdotted. Mind if I just sit and drink until it wears off?"

- Charlie Stross, "Accelerando"

Damn, I read that book as a teenager in 2006 or 07! I've been thinking about a reread whenever I remember it. It was the book that really made me appreciate the whole technological singularity idea back then.

I've been trying to find my paper copy all week so I can reread it. Might just give up and read it on his site: https://www.antipope.org/charlie/blog-static/fiction/acceler...

I just run a normal website through a normal (major company) connection with their supplied router.

I have 10 minutes or so of power outage per year, otherwise it works fine. Maintenance effort is close to zero, security update apply automatically.

I'm tinkering with a Raspi to do that sometime soon. Its going to be on an isolated network segment off an openwrt router that'll forward ports and hold the public ip. Technically it's ready to go but I'm enjoying tinkering with it still and no rush to put anything up.

its not "state of the art" but I expect it to serve my needs.

For me the state of the art is realizing that you do NOT need 99.99999% uptime for personal stuff.

Once there, do whatever you want. Get a DMZ or even a separate connection if you really want to go wild, but it's not terribly complicated.

There are even "in a box" solutions - such as https://mailinabox.email or https://owncloud.com

I don't want to open my home network to just anybody, so I have a "jumpbox" that is the lowest-end shared VM at Hetzner. It runs nginx, dnsmasq, and Wireguard; my home servers connect to it, I add other peers as I need to, and dnsmasq resolves the hostnames using Zeroguard IPs for the home network.

I have 3 sets of DNS entries for the home lab servers:

1. "internal"/home network addresses (e.g. your 192.168.x.x) 2. Wireguard addresses (e.g. 10.0.x.x) 3. public DNS entries that all resolve to the jumpbox

The purpose of #3 is to support simple Letsencrypt setup: nginx on jumpbox forwards Letsencrypt requests to the internal servers over the Wireguard connection.

Internally, I use a https://github.com/nginx-proxy/nginx-proxy setup, so that any time I want a new service running inside the home lab I just have to:

1. Pick a hostname and add it to public DNS 2. Configure its Docker container to add the environment variables that nginx-proxy looks for 3. Add the hostname to the jumpbox /etc/hosts 4. Add the hostname to internal LAN DNS

It's a little much but I like how it works. It's not so bad to get setup.

reverse dns to catch your dynamic IP changes, and port forwarding/dmz on your modem/router should be all you need if your ISP doesn't block port 80/443.

If your ISP does port block (or you don't want to port forward or expose your IP) you can also set up a cloudflare tunnel (warp?) on a server, and point it to your local webserver's port, and set your cloudflare DNS for your domain so that it points to your warp tunnel. This way requests for the website go to cloudflare then to your machine and on to your local webserver.

It depends, if you have real IP connectivity on your box you just run a web server and bob's your uncle. If you're stuck behind a nat for v4, try v6. Or port forwarding or VPN to a cloud provider or vps with better IP connectivity.

As an enthusiast but not an expert nor a person with unlimited time and money, I found that while hitting 1gb/s LAN on ethernet is easy, moving up to 10gb/s has been hard. I have 2 networking cards and a router in between them, all 10gb capable. Yet, it's really hard to get them to do that. I started with reliably getting 2.5gb/s and managed to move up to 5gb/s through some tweaks, but actually getting the full value of the hardware seems hard. The bottleneck could be anywhere as far as I know, and I don't have the hardware to swap out in experiments. I don't think it's a drive bottleneck, but even then its not clear.

So, I'm all for faster, but it seems faster comes with some headaches if you don't know what you're doing. It's just surprising that consumers are still pretty much limited to 1gb/s after all these years, and my attempt to do otherwise hasn't fully paid off yet.

I think about these headaches as extra fun :)

But about your problem - cards manage to negotiate 5 Gbps, or even negotiation fails? If it's the latter, then it might be cables.

Overall, for me, it works to simplify the network as much as possible and first achieve results on a direct connection to the router when it's nearby and then add more and more.

Oh, and also measure as much as possible (that's why I'm sad about the state of system metrics on OSX, for example)

I can upgrade from 1Gbit to 10Gbit for only 5 euros extra per month but I don't really have anything to take advantage of it. I don't think my new MacBook Air M2 even supports 6E, and wiring the apartment would be a bit of a hassle.

If you only have a few people or clients in the house, the upgrade is likely not worth it, though 10Gb would be nice to flaunt.

At that price I'd take it, but that's because I have a 10G network to my NAS, though I'd have to upgrade my router.

With the size of recent games and patches, 1 Gbit starts to feel a bit too low. :)

And about Macbook... In my current setup, I'm using Thunderbolt -> PCIe and one of the Mellanox cards; it gives excellent and stable 16 Gbps with stock firmware and DriverKit-based drivers (that comes with OSX), though it feels that DriverKit driver doesn't use any offloading that it supports (while running iperf between Macbook and router, it consumes roughly 3 cores entirely on Macbook, while the router doesn't even ramp up the single core to its boost frequency).

But you are right; rewiring is a hassle, though Cat5e usually works with 10Gbit over reasonable lengths (10-30 meters, maybe a bit more if it is perfect quality cable).

Also, even if I decided not to go for fibre, I'd still switch to at least 10Gbps because I'm not alone in that network :)

I've got AT&T Fiber's 5 gigabit symmetric service (with a Ubiquiti equipment stack that handles 40 gigabit without issue) and while its been very stable and I win all nerd fights with my speed tests, I've been thinking about dropping back down to 1 gigabit because I don't find much value out of 5. If it was cheaper ($180/mo for 5 gigabit) maybe I'd keep it but Astound/WaveG is offering gigabit symmetric for $50/mo with 4 free months so tough to swallow.

What Ubiquiti equipment are you using? Doesn’t their router top out at 3.5Gbps?

UDM Pro

USW-Pro-Aggregation

USW-Pro-48-PoE

US-XG-6PoE

Recently bypassed AT&T's residential gateway by getting a XGS-PON on a SFP+ stick, let me shave off 2ms and gain a couple hundred mbps each way, in addition to removing their crappy gateway.

Which stick did you go with?

This one [0] via a group buy on the 8311 Discord [1]. The instructions for use are at [2].

[0]: https://ecin.ca/custom-xgs-pon-sfp-stick-module-xgspon-ont-w...

[1]: https://discord.gg/8311-886329492438671420

[2]: https://docs.google.com/document/u/0/d/1UIAgtxkImgFRwyaGDGtI...

Dear lord, that is the gnarliest looking SFP I've ever seen. Thanks for the links, this looks great!

It's a full ARM stack in there. Can even SSH and telnet to it for a real shell, and it runs a web interface.

I thought it wasn't possible to bypass the BGW due to certs?

This is the XGS-PON bypass, GPON bypass is different and 802.1x (cert) based. XGS-PON just requires cloning the serial number and MAC of the RG.

The BGW350 that they give you tops out at 5Gbit on the WAN SFP interface and has one 5Gbit RJ45 LAN interface along with two gigabit interfaces.

I connected the 5Gbit to a TP-Link TLSX105 10Gbit switch which has five 10Gbit RJ45 switch ports.

Since my office is on another floor and has more stuff that needs to be hardwired, I have a TP Link TLSX1008 10Gbit switch that has eight 10Gbit switch ports.

I connected the two together and, presto, 5Gbit for everyone.

My Macs have OWC 10Gbit Thunderbolt NICs so they can each get the full 5Gbit.

I tried using a Mikrotik CRS switch/router thing, but dealing with SFP inconsistencies is annoying as hell and it was a very slow router. (You need the higher end CCR routers to get 10Gbit routing, but I just wanted to have all of my devices use the Pi-Hole for DNS, which I could do with DHCP.)

I also finally got AT&T's 5Gbit fiber service. (Interestingly, wiring up the house for 10 is easier than 5.) I've found it to be a really help for super large downloads, like bundles of container images, if you can get bits at gigabit speed from the server.

I'm definitely going to upgrade to 10Gbit if AT&T makes it available.

For home runs I'd recommend using duplex instead of simplex (the cable out to the provider has to stay to whatever they require of course). For a 100M cable it's $37.47 for the duplex and $24.48 for the simplex, the price difference of $13 for a really long run. The price difference of 2x 25BASE-LR (duplex) vs 2x 25BASE-BX (simplex) transceivers will be more like $60. Even if you want to stick with simplex optics everywhere adding <$10 more to know you have an extra run already made should you ever need it is pretty handy.

If your concern is the physical size of the cable just make sure you're getting a non-splittable duplex cable. Some people prefer splittable because then if they need a simplex cable they can just pull it apart but that's what requires the cable to be larger as you get 2 individually shielded cables. The actual fiber core is measured in microns, you can pack 100s in roughly the same sized jacket.

Bonus note: If you're ever looking at doing a bunch of runs between two spots you can look into MTP/MPO trunk cables. These put 12 strands in the same protective sheath so it's still a really tiny cable and then you can break out via a pigtail cable at the other end.

The price of 25GbaseT (rj45) is still very high. 10GbaseT (rj45) is coming down very nicely though and some motherboards, routers and high end access points have 10g ports onboard. I think it's actually cheaper to go 40gbps QSFP and run MPO cables (4 pairs internally) than it is to go 25g fiber. This is pretty much the case in datacenter right now for anything that doesn't need to go over 300M.

I think it's actually cheaper to go 40gbps QSFP and run MPO cables (4 pairs internally) than it is to go 25g fiber.

It's not. 25G and 40G transceivers cost the same and MPO cables are more expensive.

True, in terms of price, duplex is cheaper (though compared to overall router cost that is negligible). In my case, I wanted to be able to swap SFPs if input one dies (for any reason), and it doesn't make a lot of sense to buy different modules for different purposes, even if that would save a bit of money.

There is another difference - if I want to go for field connectors and cut it myself, the Simplex connector would cost about 4.15$, and the Duplex would be 6.16$, which would further increase the price difference.

But thanks for the advice, it is indeed a viable option.

The hardest part, I think, is making use of that speed.

25 GBps is about the sequential speed of a decent NVMe SSD, and much faster than anything SATA.

It is also faster than what a CPU can handle for all but the most basic of processing. Working at GBytes/s speeds require serious performance considerations when coding, often involving working with assembly, SIMD, etc... Things like the TCP stack start being a bottleneck at this point.

There is also a limit on how fast you can consume content. 25 GBit/s is about 4k@120Hz uncompressed video.

And remember that both ends need to handle the bandwidth.

Doing something useful with the full bandwidth is a challenge in itself. Running a server for many users is the most probable use case, but you need a beefy machine, maybe several to be able to serve at such speed. You need to find enough users too.

I am not saying that getting 25 Gbit/s at home is a bad thing, it is a technical achievement and I am all for it. But I feel like finding an application that can saturate that kind of bandwidth is like a continuation to the challenge. Maybe something to do with gaming or VR.

The main application here is to force myself into solving those problems :) I have plans on what I'll do after. As with a 25 Gbps router comes a 25 Gbps home NAS, then I would want to have decent latency for packet processing and I hope all of that would force me to do something interesting :)

Honestly, for most applications at home, I struggle to justify even 1 Gbit/s. At my location, I have the option to upgrade to a 1 Gbit/s connection at the click of a button. I currently get 500/500 Mbit/s for ~45€ per month. 1 Gbit/s would cost around 100€ per month.

I often think about how I could justify the increase in price for myself, but I always come to the conclusion that I just don't need it. And I am a heavy internet user. I download a lot, and I upload a lot (which is why I'm very grateful for my 500 Mbit/s in upload speed). But even with "just" 500 Mbit/s, downloading a 100 GB game takes about 30 minutes. And that's provided I have an endpoint available that can supply the full 500 Mbit/s, which is surprisingly often not the case.

If I happen to need 100 GB of data, I can wait 30 minutes.

Disclaimer: I'm talking about uplink here. Internally, between my PCs, my home server and my NAS, I'm currently looking into getting 10 Gbit/s set up. And yes, I'm sure there are some use cases out there that benefit from 1 Gbit/s uplink. But I, as a power internet user, have decided for myself that I just don't need it, which means that the average Joe definitely doesn't need it.

Still, I think it's great that it's available here and I'm all for getting that expanded further.

I think this shows a mindset difference in how we think about net connectivity vs other capacities. For things like power grid connections, car horsepowers etc, we like to size things so that we don't max them out very often.

For eg a 150 GB game download on your console, a 25G connection sounds useful.

And of course a fast connection also enables new kinds of applications when enough people have them (eg game consoles could use networked on-demand storage for assets so wouldn-t be size limited by console SSD capacities). As long as bandwidth is very unequal between users, the faster end doesn't get applications built for it. That's why we should enact policies that make fast connections widely available. Fiber is cheap after all.

I have 10Gbps via Sonic.net and n Unifi UDM SE. I don't have 10Gbps internally, though, but the speed test on the UDM SE got about 3.5Gbps. I get 900+Mbps internally and torrents can do about 80-90MB/s. It is glorious.

An interesting note about the torrenting: I was shocked that my Mac Mini (Intel) running 12.$latest absolutely could not keep up with more than 20MB/s. The network stack just broke and the machine would actually stop responding to pings.

I moved it to a Linux box and ran Transmission (same software) in a container and it is where the 80-90MB/s numbers came from. Doesn't phase the box it's running on.

Seriously disappointed in the Apple networking stack.

If you want to know more about init7s network and how they can provide 25gbits see this presentation they did https://youtu.be/wXmJCzMeIBo

After reading your article, I've started to think if I want to get myself a splicer :D Before, I felt that field connectors would work just fine and splices were too expensive...

The most beautiful part of it is that no matter what fiber plan you pick, the price is always 777 CHF/year.

This effectively means that the only thing stopping you from getting a 25 Gbps connection is, well, the fiber connection and the hardware

Man, and I thought I was cool for getting my home configured for 10 gigabits.

It wasn't too hard or expensive to get used 10 gigabit cards for my router and server. It was a bit pricier to get a used Thunderbolt 10 gigabit adapter for my Mac, but still not prohibitive.

I thought I was really future-proofing by making my house 10 gigabit compatible when I only have 2 gigabit internet, but if I have to worry about 25 gigabit internet everywhere I think I think I give up!

The cool thing about fibre is that it is way more future-proof than good old copper.

By the way, I also went for a Thunderbolt enclosure and network card for Mac. It is a bit pricy (the enclosure costs way more than yet another ConnectX).

I have been on 7Mbps / 1Mbps until 2021. Moved in with GF, she had 100Mbps ADSL which blew my mind. Now we have 2.5Gbps FTTH and I feel blessed. I'm moving to a slightly remote place soon which has 200Mbps FWA, and still it blows my mind. I can't imagine having 25Gbps at home. Insane.

Downloads matter up to an extent. What still gives me day 1 shivers is 600Mbps upload after years at 1, and sub-10ms latency.

At some point in my life, I was forced to use 7/0.7 Mbps ADSL after having 50/50 Mbps and that ADSL was very oversubscribed (high packet loss and huge speed drops to <2 Mbps during weekends or evenings). Later on, I had FTTH, but I couldn't control the router (fibre was terminating at the landlord's house, and I had cat5e from the router to the place where I lived, and the landlord kept the router in the worst place possible so it was overheating during the summer).

Since then, I have tried to be in control of network hardware, and I now avoid *DSL, even though VDSL is decent nowadays. But the reason for 25 Gbps for me is "just because", as ISP charges the same monthly amount as I pay now (there is a setup fee for 25, though).

I remember going from 56kbps dial-up (getting about 5-6KB/s) to a T1 and downloading an mp3 went from 20 minutes to 30 seconds. Diminishing returns past that point for sure :)

> The top shows that single-core performance is the bottleneck, and all CPU is consumed by ksoftirqd.

Trying out VPP instead of Linux kernel networking might make some sense: https://fd.io

It will likely reach the target performance, although the NIC requirements need to be rechecked.

But... well, this software is not for the faint of heart ;)

You can/should also use hardware offloaded queues wherever possible, or look into kernel-bypass networking.

Well... I did more than I've described in the post (I'm the article's author). It just became too large, so I've decided to stop and publish a second part once I run more tests and fix some problems.

So far, I have yet to figure out all the problems and settings with Netronome (or give up on it). Still, it is Netronome-specific as I can easily get line rate over TCP single stream with Mellanox cards, and the CPU doesn't even break a sweat (however, I have relatively simple routing rules).

I want to try out programmable NIC, and as far as I understand, Netronome has some support there; I just wanted to get a simple setup running first and then optimise/improve it. I tend to perfect the setup, and if I let myself do more and more and more - I will always need more time to finish it.

This seems like a bonkers solution to be putting packets through a 150w CPU, especially with the price of electrity in Switzerland.

It's putting a packet through a 150w CPU like weighing a brick is using using a 500 lb scale. I.e. just because it can do that much and you're using that device doesn't mean it's the relevant number to look at. When you're not pushing things it'll be near idle wattage, when you are pushing things like mad it'll be around <4 core wattage (which will still be a lot less than 150W) in this workload. The important thing is it's a CPU which has a great single thread performance and, as a result, it has a high peak multicore wattage you won't actually care about.

Overall I'd bet all of the NICs and transceivers are the majority of the power usage in a given year.

I still need to measure and optimise power consumption correctly, but it drew about 70W from the socket during the tests, according to my simple power meter. CPU never ran at boost frequency, and a lot of other stuff is in a low power state.

I'm a proud owner of a 1 Gbit/s fiber connection with a 100 Mbit (wired) network in my house. I want to upgrade it to match, but I don't want to go through the effort because I never run into download times for anything.

OG Cat 5 (non-e) I take it? That is such a pain to redo, I rewired some of that myself and it was a drag. If my job weren't being the literal network guy I probably would have stuck with just using Wi-Fi too!

I have 10gbit service, 1gb internal network. So like you, but 1 generation forward.

I don't usually saturate 1gbps, but I definitely saturate 100mbps with XBOX downloads, any cloud provider, torrents, etc.

You could probably pull your old cables fairly easily. Attach two wire pull tapes or strings to the old cable, pull the old cable out with plenty of wire pulling lubricant, and then use one tape/string to pull a Cat 7, leaving a spare for mistakes or additional pulls.

Do yourself a favor: buy some fiber cables, two switches and connect both ends like that. This is the only way to "future proof" it (past 1 Gbps) and will avoid you pulling cables for a while - I hope.

You can make it work with 2.5GbE and maybe even 10 GbE with some luck. But looking at its current state and price I personally would agree that fiber is more future proof.

I also think that fiber (per meter) is more cost effective than copper anyways for home use, so for me it's a no-brainer

I get about 200Mb/s on my 1Gb/s connection because my office is at the other side of the house to the router so I use mesh WiFi. Given that all my walls are solid brick, that's not too bad, really.

I'm so jealous of this. I live in a major metro area of NYC and there's still no fiber out to my town. This is so cool.

I have 1Gb/s at home. Somehow it doesn't translate into a reliable streaming service experience. I'm currently subscribed to Disney Plus, and trying to watch tv shows via Chromecast rarely works. I even upgraded to the latest Chromecast to try to fix the problem, but it didn't make it much better. I'd trade for a 20mbit/s connection if the streaming services would send me content instead of "an unknown error occurred"

How many hours can you saturate that 25 Gbit connection up and down before support calls you and tells you to knock it off?

I ran about 8TB on my fiber in one month, and no ISP notices to speak of. That's only ~25mbps sustained. You could easily push 500+TB on a 25gbps connection if you were doing something naughty with it, and I'm sure you'd hear from them about it if you did!

I used 2TB on my cable modem once and got called with them trying to upsell me to Unlimited data for $75/mo more.

I regularly use 5 - 10 TB on my fiber each month, and have never been contacted.

You absolutely should not ever get hassled for using a product that you already paid for, that's incredibly stupid. Unfortunately the US just treats anything that isn't a corporation like trash.

It's not necessarily the premise of a service with disjoint bandwidth and transfer amounts that's the problem IMO. Separating bandwidth and usage provides great value to most people who would otherwise have to pay for a much more expensive service if the two were always locked together literally. Where we could use better regulation is making the fair-use policies more upfront and useful so you don't have to dig for them or try to guess what they mean.

In my case, init7 defines that "fair use" is 0.5 PB per month. So that is about 40 hours until they start asking questions, I guess :)

I doubt I will be any close to that.

Your account seems to be shadow banned (or not enough karma?). I vouched for this comment and your other comment so other people can read it.

Thanks.

Yeah, I have something like 12 karma or so, and the account is newly registered (a few days ago). Unfortunately, HN doesn't seems to provide any workarounds for that.

Crying here with my 200 Mbps fiber. Paying $600 a month even though we have 24-port fiber coming in the premise.

Spectrum sucks

I would like to know more about this. What town? Why so expensive?

WiFi

While I love that ISP are continuing to ramp up the speeds beyond 5 Gbps, keep in mind that 99.99% of home user connect on WiFi which taps out at like 1-2 Gbps.

Reply on news.ycombinator.com