Hacker News Reader

Comment by jiggawatts | Hacker News Reader

Some people are saying that they don't feel the need for > 1 Gbps in a home network.

As a counter-point, I'm regularly limited by 1 Gbps PHY limits. During the 2 years of the pandemic I've been working from home with gigabit Internet. Why gigabit? Because that's the "speed limit" of the Ethernet PHY in the fibre broadband box, and also my laptop's ethernet port.

My laptop can write to disk at multiple GB per second, or tens of gigabits.

I transfer large volumes of data to and from a "jump box" in Azure that can easily do 20 Gbps out to the Internet.

I regularly update multiple Docker images from scratch that are gigabytes in size, each. I then upload similarly large Docker images back to Azure.

Even when physically present at work, I'll regularly pull down 50-200 GB to my laptop for some reason or another. One time I replicated an entire System Center Configuration Manager deployment (packages and all!) to Hyper-V on my laptop because the NVMe drive let me develop build task sequences 20x faster than some crappy "enterprise" VM on mechanical drives.

I have colleagues that regularly do big-data or ML queries on their mobile workstations, sometimes reading in a decent chunk of a terabyte from a petabyte sized data set.

All of these are still limited by Ethernet PHY bandwidths.

Note that servers are 100 Gbps as standard now, and 200 Gbps is around the corner.

The new Azure v5 servers all have 100 Gbps NICs, for example.

View on HN

A lot of that sounds pretty incredibly wasteful and I don’t think it should be something to strive for more of.

Docker images are layered for a reason. If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong.

Azure charges (like the other major cloud providers) by egress data. I feel bad for whoever is paying the bill for 20gbps going out.

In general, people in ML shouldn’t be pulling raw data in the hundreds of gigs down to mobile workstations. That’s a security and scaling disaster.

Trying not to be too much of a curmudgeon here, but this is the problem with 1 gbps connections. They are so good they prevent you from doing the right architecture until it’s far too late.

A good remote work setup should leave all of the heavy lifting close to the source and anything that does need to be downloaded should be incremental/shallow copies. You should be able to work from a stable coffee shop WiFi connection.

> If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong

When I worked at Vercel during the days we supported Docker hosting (back when it was still ZEIT), we saw that even among many tenants on the same machine, the "layers" thing wasn't really that beneficial - we still had to fetch a ton of stuff over and over again.

So this is great in theory, but it doesn't really pan out in practice.

I believe Docker has even throttled/blocked corporate users doing excessive layer pulls from Hub (and suggested they sign up for X plan).

Reasonable, IMHO.

That's a recent thing, as I understand it. After Hub switched to their enterprise plan setup.

ZEIT had a pretty aggressive cache anyway, for the short time it was implemented (before ZEIT moved away from docker hosting, at least).

I always thought it was a little strange given that layers are easily CDN-cacheable indefinitely. That said I'm sure they're using something like a CDN but even the cheapest bandwidth gets expensive at level of scale.

"You should be able to work from a stable coffee shop WiFi connection."

I think there are different philosophies, just like 'no comments allowed in code', this is a matter of judgement.

I saw 5 different corporations setup centralised workflow, and usually you end up with a massive cloud bill, incorrectly configured admin rights, under-resourced VDI, and a ticketing system that takes a week to resolve the smallest issue.

If we assume for the moment that data in question is not highly sensitive (for example public satellite imagery), there is nothing wrong with getting every dev a beefy workstation with 50 TB of storage and a 20-core CPU, and it's often even cheaper.

Then again, if it is public (or not-sensitive) data, why make it part of your infra?

If you are working with ML and have public datasets, just use the original source. Make a public mirror if you want to collaborate. Make it accessible via BitTorrent. You can bet that the amount of savings you will get from AWS will be enough to give each dev a NAS with a few tens of TB, which then they can even then use to host/mirror the public data by themselves.

My specific example was a private data set hosted on-premises (satellite imagery).

The analysts were pulling down a ~0.1% sample to develop the ML algorithms locally, and then deploying the final thing in the cloud to run through the whole petabyte.

Local development has its advantages, such as high flexibility and very fast cycle times. It's fantastically expensive to outperform a modern PCI 4.0 NVMe drive in the cloud. Locally it's a $200 part.

If it’s on-premises, why are you talking about cloud costs?

5? Are you a consultant who goes in to fix these situations?

> Docker images are layered for a reason. If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong.

Well the images need to be in some order, and if you are fine tuning the very initial parts of the image, this can happen.

But it seems a better transfer algorithm could be used for those individual layers. It rather seems like if they don't match, everything's transferred, while there could be a lot better options as well. If we consider that they are structured like file systems then it opens even more options.

---

So I took a peek on my local Docker installation. It seems the layer files on the drive are gzipped json.

That's right, .json.gz, _that includes the files BASE64-encoded_.

Yes, I believe there is room for improvement, if this is also the format they are transferred in.

base64-encoding data only poses a 1% overhead[1] when the data is subsequently gzipped, so this is hardly an issue in practice.

[1] head -c 100000 /dev/urandom | base64 | gzip | wc -c

I wasn't really consider space overhead, which as you notice is small, the encoding just something I noticed. Just json+bloatified binary+compression seemed the epitome of solving problems with a hammer when you know the hammer really well.

But as commented to me, that was incorrect, the payload in the files I saw was actually binary-encoded metadata. The actual archives are stored only in extracted form in the client and .tar.gz in the registry. Not that putting binary metadata in JSON is really optimal to me either, but maybe there are terrific reasons for doing it. Just check /var/lib/docker/image/overlay2/layerdb/sha256/*/tar-split.json.tar.gz; the fragments seem to be straight from tar, so maybe tar is used to interpret them?

Certainly .tar.gz files are not the most optimal for the purpose of incremental updates either. It's difficult to say how much a more sophisticated transfer protocol/storage format would benefit registry servers, though some clients could benefit a lot from it.

If a space-efficient merkle-tree-hashed archive-format suitable for efficient incremental updates doesn't exist, then someone should make one ;-).

json is used only for manifests; actual layers are .tar.gz.

You are correct; I too hastily took the "payload" in the tar-split.json.gz to be the binary itself, but it's something else.

Nevertheles, an archive format other than .tar.gz could be transferred more efficiently given a previous version of the same image. Or, even .tar.gz could be transferred more efficiently if both ends extract the file first, and if there is a need to preseve sha256, then that needs a bit more work.

I also recall the saying "you can't save yourself rich"

There are some times when your thinking/workflow is constrained by your environment, sometimes significantly.

But the person is already super rich and has developed such an obscene yacht sinking habit that they are still money constrained.

As a counterpoint: I had 10G networking and went back to 1G last time I reorganized. Haven’t really missed the 10G for now.

I ditched the 10G because the 10G switch was hot and noisy and had a fan, whereas my 1G equipment was silent and low power. This could be overcome by putting the switch in another room, obviously, but I haven’t missed it enough to go through the trouble yet.

> During the 2 years of the pandemic I've been working from home with gigabit Internet.

Doesn’t this make every other point in your post moot? I’m similarly limited by Gigabit internet, so the only time I benefit from 10G is transfers to/from my local NAS. I realized that I almost never lose time sitting and waiting for NAS transfers to finish, so even that was only a rare benefit.

If I was buying new equipment and building out something new, I’d pick 2.5G right now. It’s significantly faster than 1G but doesn’t come with the heat and noise that a lot (though not all) of 10G switches come with.

I’m sure I’ll go back to 10G some day, but without >1G internet and no local transfer use case, I found the extra hassle of the equipment (laptop adapters, expensive/hot/noisy switches) wasn’t really worth it for me yet.

This is exactly where I ended up recently. I went from noisy and power hungry Cisco/Juniper 10G switches to a silent Ubiquiti GigE switch with PoE and haven’t been happier.

10G was cool but I can’t recall a single instance where I really wish I had it back.

> 10G was cool but I can’t recall a single instance where I really wish I had it back.

Same. It only really made a small difference to my most transfer-heavy workflows. Even a fast NAS is still slow relative to local storage (it’s not just about sequential transfer speeds).

I’m sure I’d feel differently if I was doing something like video editing where I had to move large files back and forth frequently and wait for them to be ready.

All of my 10g equipment is silent or near silent. Most recent purchase is the TP-Link TL-SX105. Take another look the next time you redo your home network. At least consider multi-gigabit.

I am on FIOS 1G home fiber, how does one get 10G ? Do they have run a line from the node or street ?

If your provider offers 10 Gbit/s, you only need to change the fiber optics modules at either end of the connection. The same lines can be used, no need to run anything extra.

I ended up buying one of these for my 10g home network. It's completely fanless. The 1G Ubiquity EdgeRouter-X I have runs way hotter.

https://mikrotik.com/product/crs309_1g_8s_in

I don’t see a use case for 10gb Ethernet in your home network in your post, unless I’m missing something? It sounds like your ISP is limited to 1gbps and all your use cases seem to be bottlenecked by it.

Do you have home servers you forgot to mention that you’re uploading docker images to, and those could benefit from 10gb Ethernet?

(For me, I use gigabit Ethernet everywhere at home but sometimes need to transfer large disk images from my desktop to my laptop, and using a thunderbolt cable as a network cable helps here, I can get closer to 2 gigabits of transfer speed before disk writes seem to be the bottleneck.)

In my case I'd like to transfer from/to my NAS where I centralize a number of data pieces.

I work inside a workstation but occasionally on a laptop as well.

Having 10GbE will help me when I'm on my workstation. It's not fatal, definitely, but it adds up to lost productivity.

Is your NAS NVME based if you are using spinning metal you won't reach speeds to saturate 10GbE especially with small file sizes?

It's a ZFS-based NAS with an SSD cache. Right now it's only 1x SATA3 SSD which means it will top at 500MB/s (so slightly less than a half of 10GbE) and I am planning to add one more which will make it almost saturate a 10GbE link.

Additionally yep, I plan to have one NVMe dataset sometime this year.

And even further, I plan to add a few enterprise HDDs in stripe and mirror. If you have 8 (with capacity of 4 for full mirroring) and each is at 260MB/s then et voila, you get 10GbE again.

If somebody knows why they should pay a premium for 10GbE, they also know how bad spinning rust is.

Those same people should also know that spinning rust can go very fast as well... My 8 and 12 drive spinning rust pools can read at 500MB/sec easily and get close to GB/sec.

But the 10Gbit network is there for the flash pool.

Yeah, but... The noise! I just can't handle spinning rust anymore for this alone even if the $/GB ratio is still so much better.

Consumer SATA SSDs are plentiful enough that I'm just going all in on flash when I revamp my server and do an 8 drive (+1 hotswap) RAID10 build or whatever. I don't need +60TB or anything and 8-9 drives is about the limit you can find on high end consumer mobos I think...

> The noise

It is not required for the NAS to be in the same room where you sleep.

With a proper drives (and maybe some APM tinkering) you can make a very quiet one. Also using 2.5" drives is an option too.

Though if you can throw money at the problem than of course you can just buy a bunch of TB SSD drives.

Or say 1Gbps is too low bar. Even single rust drive can fill 1GbE in simple workload.

Put 8 high-quality HDDs in a stripe & mirror config. Thus you can read from 4 simultaneously. They each top at 260MB/s. That amounts to little over 1GB/s -- which is 10GbE.

I have a Linux NAS with 6 7200RPM HDDs in a raidz2 ZFS pool, and the local fs read speed on large files is 7 times faster than GbE, about ~930 MB/s. So while this wouldn't saturate a 10GbE link, it would still be greatly beneficial to multiply the network read speed by 7-fold by upgrading from GbE to 10GbE.

How did you bench your ZFS pool?

With dd. And normally when I do this I drop the caches with "echo 3 > /proc/sys/vm/drop_caches" but this doesn't work with ZFSOnLinux as it has a separate ARC cache that doesn't get cleared by that command (and I don't believe it's possible to clear it, other than by exporting and reimporting the zpool.)

So I read a 10GB chunk of a file that I know is not cached:

  $ dd if=/some/large/file of=/dev/null bs=1M count=10240
  10240+0 records in
  10240+0 records out
  10737418240 bytes (11 GB, 10 GiB) copied, 11.3378 s, 947 MB/s

The 947 MB/s figure printed by dd matches the speed reported by a "zpool iostat tank 1" running during the benchmarking, which confirms I'm not reading from the cache. When I repeat the same command, dd completes almost immediately reporting a speed of about 6-8 GB/s on my machine, while the zpool iostat command shows zero read bandwidth.

More devices used at once? If I am downloading 1 Gb/s from the internet, and meanwhile one of my housemates want to look at a movie from our NAS, the other one wants to backup 100 GB of photos to the same NAS - then 1 Gb/s home network is not enough.

4K video doesn't require high bandwidth.

Doesn't matter when the entire network bandwidth is taken by my 1 Gb/s download and/or the photo backup. Everything on top of that needs too much bandwidth then.

1 Gb/s network is probably enough for most people, I agree. But I certainly think there are many use cases for faster networks, especially in digital-heavy households.

Indeed, a 4K video can hardly saturate 100Mbps link https://www.reddit.com/r/PleX/comments/eoa03e/psa_100_mbps_i...

I am confused, as the link appears to say the opposite.

> Conclusion:

>The majority of 4K movies (75%) I tested have bitrates over 100 Mbps and many seconds where bitrates spiked over 100 Mbps. Some have 100s of seconds where bitrate spikes over 100 Mbps, and will most certainly cause problems if played with bandwidths less than 100 Mbps on devices that don't buffer well such as the LG TV or Roku TV. To make sure you get the best experience without any buffering or transcoding on such devices, you need to make sure you have a bandwidth that exceeds at least 150 Mbps to play most 4K movies properly. Ideally, it should be higher than 200 Mbps.

The highest average bandwidth shown was 73 mbps. You probably need 150mbps to comfortably play 1 4k move, but once you are looking at the effect 4k movies have on higher bandwidths, average bandwidth becomes more relevant. You could pretty easily stream 10 4k movies over a 1gbps channel since the odds that all of them will be over 100mbps at the same time is low (and even if it happens briefly, it will be handled by buffering).

> certainly cause problems if played with bandwidths less than 100 Mbps on devices that don't buffer well such as the LG TV or Roku TV

"If" is doing some heavy lifting there.

The linked post shows that the average bitrate of every sampled 4k movie was less than 75 Mbps. The author even bolded "on devices that don't buffer well such as the LG TV or Roku TV"

I have Jellyfin setup and there are times when 3ppl would watch something. My entire collection is the highest quality I can find on the net so normally a movie would be around 80-100GB.

Plus I have a service which downloads stuff for the archive team so that’s always doing some network traffic.

There is also a CI gitlab worker and that is also always doing some build with docker images from scratch.

I just wish more than 1Gbps was something that was offered and I can upgrade but so far I’m limited by my ISP with no way to upgrade. Inside my network I have 10Gbps and I have never hit that limit. It was expensive and I needed it for a now deprecated servicing.

I'm not arguing against faster networks, but scenarios like "one fast download makes video streams buffer" can be solved by using better routing algorithms (CAKE for example) instead of making the pipe so wide that it'll never be close to full. One of these is a configuration flag that you can flip today and costs nothing, the other means upgrading infrastructure.

What is CAKE??

https://www.bufferbloat.net/projects/codel/wiki/Cake/

Your 1G switch should be able to do 1g from your computer to your internet router and 1G from your roomate to the NAS and whatever from your NAS to the guy watching movies. Even cheap gigabit switches can process (large packets) at line rate on all the ports. If your NAS is also your internet router, maybe you can't make it work with a 1G switch, unless it can do link aggregation.

For a moment I thought about setting up 10GbE at home, but then I realized there is no point yet because my external connection to internet is limited to 1 gigabit.

The only use internally would be central storage server. But all the new NVMe SSD's are faster than 10 gigs, so my internal network would be the bottleneck.

The next logical step would be upgrading to 100 gig network, but that is too expensive right now. I should buy more fast NVMe storage for that money.

If I could get 10 gigabit connection to internet, it would probably make more sense, but I don't see it coming that soon. And I'm paying 15€ for 1 gigabit right now, I would not be willing to pay much more for 10 gigabit connection.

On top of that, my Zabbix monitoring data shows that I rarely saturate the 1 gigabit connection to internet. Most of the time I don't need more than 10 megabits.

Oh, and I would never actually use the copper cables with RJ45 for anything above 1 gigabit. Those ethernet cards heat up a lot, use too much energy. This would be a bad investment. Get a proper patch or optical cables and use SFP+ instead.

I’ve never had this as a problem, and I use 50-100ft of Cat5e + 10GBaseT SFP+’s… it’s well within the thermal limits of the SFP, which probably uses 2-5W.

I'm surprised you're able to get anything reasonable for 10G over any modest length of cat5e, i made a few attempts at doing it for fun when i was upgrading some of my stuff and could only get anything to work with cat6 at a minimum for more than a few meters.

What I found was that the bottleneck was often my home firewall. Plenty of bandwidth inside, plenty to the cable modem…but using a Cisco or Palo SOHO firewall of hand capability was more than I wanted to spend…sure I could have set up a VM environment and run a virtual firewall…but I didn’t really want to. I’d done that in the past and talking the wife through a safe VM environment cycle when away from home and something went wrong was…harder than telling her to power cycle a netgear product.

A few months ago, I bought 2 10GbE NICs off Ebay to connect my main desktop and basement server directly. Since then, I've never seen a transfer between them that was slower than 120 MB/s, usually about 160-200 MB/s. (no RAID or NVMe to NVMe transfers)

Some advice: make sure you know the form factor of your NICs. I accidentally bought FlexibleLOM cards. They look suspiciously like PCIe x8, but won't quite fit. FlexibleLOM to PCIe x8 adapters are cheap though.

How do you regularly pull 50-200 GB? This sounds like terrible practices or implementation at work.

I've never seen docker images of that size that can't be shrunk with multistage, or docker shrink. I can see how someone could get to that size with just baking in all their models to the image itself but yuck.

windows docker images are incredibly bloated. I don't think I've seen one that size, but the base image for a windows container is like 17GB by itself, if I remember correctly. Basically just pulling down an entire Hyper-V image.

> windows docker images are incredibly bloated

They are no-where near that size ...

base image for .NET Framework is 10GB.

they only get larger from there. Largest on my machine right now is showing to take 20GB. Granted there are a lot of shared layers between images, so the amount of space on-disk is much less than 20GB. But if you don't have any of the layers already? You're looking at 20GB easy.

And you have to start from scratch each month if you want to keep up with Windows Updates -- they apply to the base image.

This isn't true...

I just downloaded: https://hub.docker.com/_/microsoft-dotnet-framework

It was about 2.7 / 2.8 GB in size.

Total size on disk was 8.7GB but we are talking about download here.

I would say you should be moving to .Net 6 it's been several years now that .Net Core has been out so there is no excuse.

None of this is an excuse to get a 10Gbps internet connection to your house. I have a 500Mbps connection and with the corporate firewall my laptop gets 100-200Mbps connection and pulling those containers was like 10 minutes.

Edit: In case anyone thinks I'm lying:

https://devblogs.microsoft.com/dotnet/we-made-windows-server...

> As a counter-point, I'm regularly limited by 1 Gbps PHY limits. During the 2 years of the pandemic I've been working from home with gigabit Internet. Why gigabit? Because that's the "speed limit" of the Ethernet PHY in the fibre broadband box, and also my laptop's ethernet port.

same, I live in a rural village in france where I get 2GB fiber. Yet all my computers aren't able to leverage that.. quite frustrating to see speeds being limited at 110/120 megabytes/s where my drives are easily able to handle gigabytes.

> Note that servers are 100 Gbps as standard now

In what world?

Yeah, I had a hell of a time trying to get our internal network to 100Gbps and was told by multiple experts I was being too ambitious. I ended up with 100Gbps between switches and 40Gbps to the servers.

In the world of unlimited budgets, where employees all spend their days pulling gigabits per second of egress from the cloud, apparently.

100GbE NICs are <$500, and 100GbE switch ports are $100-150.

while 10G is now at sub-$30 range, totally accessible to everyone )

Could you recommend a switch and some NICs, along with a thunderbolt adapter?

Switches: when I last looked, there were cheap mikrotik switches and tp-link, if you want to buy new. Otherwise search on eBay/craigslist/etc.

Network cards: intel, mellanox, solarflare. Intel has a lot of drivers for all OSes, mellanox is most stable electrically (you can provide it with separate power, no need for fancy adapters, just plug any 12v in), solarflare is cheapest on ebay if you buy 2-3-generations old cards. Actually, any 10G network card will do, just check if they have drivers for your OS.

Thunderbolt: there are ready-made eGPU cases, they are costly but easier to handle. If you want to be cheap: wait for sale of any thunderbolt3-to-nvme adapter. PCBs for all of them are made by the same factory (you can find it on alibaba/taobao) they differ only by box and branding. Based on intel JHL6340

edit: I have got this, it was on-sale at the time https://www.aliexpress.com/item/4000200151507.html

edit2: just checked eBay, the cheapest solarflare "S7120" (it is SFN7022F actually) is $15 (!) + free shipping, wow. Though you'll need some sfps, they start from $6 (FTLX8571)

How much did you pay when your tb3-to-nvme was on sale? It's not an urgent purchase :)

$55 + shipping (less than $5)

I use a map-maker app for creating game levels, the data is stored on a central server when it's saved.

A 4kx4k map layer section, in RGBA is 64MB. Each layer on the map has multiple types of mask applied (brushed alpha, brushed noise-function,...) so there's another 64MB of XXXX data per layer. A map can have (say) 10 layers on it, so we're up to 1GB, and then there's the data for ephemeral layers ("stamped" images at co-ords and scale, alpha,rotation). The stamps themselves are usually stored in an 8kx8k layer so that's another 256MB

When your 'save' action is 1.5GB of data, a 10Gbe network is much preferred over a 1 Gbe network...

All of this is rendered down for production of course, but during development, each layer is kept separate along with its masks.

> Each layer on the map has multiple types of mask applied (brushed alpha, brushed noise-function,...) so there's another 64MB of XXXX data per layer.

Wait, why are the masks RGBA instead of grayscale?

8 bits per mask, more than one mask, packed into a 32-bit value.

1st one is typically a brush-in one - basically just Alpha brushed in from various alpha-brushes over the map

2nd can be a noise function, where the 1st alpha mask is multiplied by the second to get a brushed noise, and the noise might be useful for randomly-shaped patches of different-colored grass in a scene

3rd might be a shadow mask, where any objects "stamped" on top apply a default shadow at a given angle, and that default shadow is written into this channel. Then the map-maker can alter those if (for example) a shadow hits a wall.

Then there's the shader mask - so you can select a given shader for a given area of the map. These'll all be batched up later, but if you want flowing water, brush in where you want the water shader to be applied. Same for smoke, lava, etc. etc.

OpenVPN has support for compression. Makes me wonder if it would work in your case.

I went to 10GB ethernet (and 2GBps Fiber to the home) and I couldn't go back. It is amazing. I did have to upgrade the NAS to be able to handle consistent >1GB streams in and out though.

> Azure that can easily do 20 Gbps out to the Internet

I have tested Azure and AWS speed out to the internet and they too won't get 20/100 Gbps. That high bandwidth is for connecting instances within the AZ. Once any of the traffic has to go over the internet it's much lower.

> I'll regularly pull down 50-200 GB to my laptop for some reason or another

This is a serious failure in your workflow. You should be looking at reducing that size considerable in terms of a docker image.

> AWS speed out to the internet and they too won't get 20/100 Gbps.

Exactly. I understand why some may want or need 10Gbps. But OP comment should not even be top voted.

It depends on context, but we solved this at work with virtual desktops through VPN. The heavy lifting is done on-site, the users have a thin client that barely exceeds one Mbps.

Yes, virtual desktops suck in several aspects, but we couldn't expect everyone to have gigabit Internet at home.

I've been ranting about this for a while tbh. Network interfaces used to be one of the fastest interfaces on a computer. Now they're one of if not the slowest save for USB2 or bluetooth.

1GbE networking is one of the slowest interfaces on modern computers.

- The slowest USB3 spec is 5Gbps. USB3 is a mess, but you can easily far exceed 1Gbps with the absolute worst of it.

- The current fastest USB spec is USB4, and is at least 20Gbps, and sometimes 40Gbps

- Thunderbolt/Lightning both push 10's of gigabits depending on spec

- 802.11ac ("Wifi 5"/"Wave 2") is capable of Gigabit speed, and 802.11ax ("Wifi 6") easily exceeds 1Gbps

- Even Cellular connections can exceed 1Gbps these days with 5G, though admittedly realistically you're looking at 100Mbs-1000Mbps

- Individual SAS/SATA rotational drive easily push passed 1Gbps in seq workloads

- Individual SAS/SATA SSD's often push 3Gbps, and SATA itself supports up to 6Gbps, with SAS up to 12Gbps.

- PCIe NVMe in it's slowest spec (Gen3 at 2 lanes) is still 16Gbps, most typically running at up to 32Gbps at Gen3 x4

- PCIe5 consumer devices have launched (Gen5x4 lane NVMe is 128Gbps)

- PCIe6 is around the corner for enterprise

To add insult to injury, in the consumer space they're trying to tout 2.5GbE as some sort of premium new NIC, despite it replacing 10GbE in the premium NIC space from several years ago (eg Asus X99-E-10G-WS). It's maddening. 10Gbase-T, 10Gbe, and Cat6a has been around for well over a decade (to say nothing of SFP+ 10G). Granted they're starting to reposition 2.5GbE as the new 1GbE, which is far more sensible, but still positions your NIC as one of your slowest interfaces

1GbE is the limiting factor to speeds these days, with internal storage/devices and nearly all alternative interfaces vastly outspeeding 1GbE. Even cheap external SATA SSD's and a common usb3 cable can greatly exceed your 1GbE network speed. If you have multiple devices, and want to transfer data between them, a network cable is supposed to be the superior choice. Not Wifi, Not USB, but a network cable over the local network. But it's actually much faster to plug a drive into one device with USB, transfer the data, unplug, plug into the other device, transfer again, than to use a standard 1GbE network interface. Shameful. Just because your WAN interface is likely 1Gbps or less doesn't mean there isn't a use at the prosumer level for >1Gbps network connections for use at the LAN level.

Ethernet is by far the longest connection out of all of those, supporting up to 100 meters for 10G over copper and over 300 meters for fiber [1] which is pretty impressive. No other commodity interface comes close to that kind of range at that speed.

Wifi 6 officially has a real world limit of 700mbps [2] and drops to half that if you're more than a few feet from the router. Passive USB 3 and thunderbolt cables have a max length of 2 meters [3]. The rest of the interfaces you've listed like PCIe all operate on the scale of inches and plug straight into the motherboard.

Plus it really isn't very expensive to upgrade to 10G given the use case is pretty niche for home use. You can get a budget SFP+ Switch and 2 NICs for ~$300.

[1] https://en.wikipedia.org/wiki/10_Gigabit_Ethernet#10GBASE-SR

[2] https://www.theregister.com/2019/12/05/wifi6_700mbps_speeds/

[3] https://www.newnex.com/usb-cable-maximum-length-limits.php

Yes. The connectors have different lengths, as some are internal and some are external interfaces, both of which are part of the chain in transferring data from storage device in one host to a storage device in another host. The point was to illustrate the bottleneck among all of the possible pieces in that chain, standard 1GbE ethernet is lagging behind. That's why I specifically referenced NVMe and 2/4Lane implementations when talking PCIe. In this context, run length is entirely irrelevant. It's more relevant to point out that you can network Ethernet connections, meanwhile USB, thunderbolt and Wifi are primarily client <> host connections.

> You can get a budget SFP+ Switch and 2 NICs for ~$300.

Yes, this is part of the problem. In 2016's we started to see mainstream adoption of 10G, which hopefully would lead to the more availabile, more affordable, consumer friendly ease of use twisted pair Cat6a and RJ45 10Gbase-T Switches. Instead, Enterprise had already largely moved on from 10G (largely supporting it as a legacy connector via SFP28 being backwards compatible with SFP+) and Mainstream/Prosumer manufacturers instead decided to regress from the 10Gbase-T deployments to 2.5GbE.

As a result we're not seeing a lot of development in the 10G space, and in many ways a regression towards 2.5GbE. Sure, if you don't mind the noise and know your way around a cli, there are plenty of surplus enterprise switches on the cheap. And I don't mind Optics/DACs. I run 10G at home (cat6a, Intel X550's, Arista 7050T-64, which used to be a few hundred bucks) and 25G/100G in the lab (CX4 LX's/CX6 VPIs connected to an Arista 7260CX-64). But it is unfamiliar to most consumers. Not a huge hurdle, but a strange ask considering 10Gbase-T with familiar twisted pair and rj45 aint new by any means. Mikrotik and UBQT both have some options in the space, but the cost of these newer switches certainly doesn't fall in the $300 budget, unless you only need like 2-4 10G ports on the switch and are fine with SFP+. And yes, >4PHY 10G SFP+ and/or 10Gbase-t switches exist, but you're not getting them and NICs w/ DACs (or especially fiber if you need a longer run akin to a standard Cat5 deployment) inside your $300 budget, and there's very few on the market that'll fit in your budget even if you stretch it to $500.

TL;DR: Yes, 10G is technically available, but it's a strangely esoteric niche that exists after the mainstream decided to regress from 10GbE towards 2.5GbE.

awesome dude, I have a 10Gbps setup at home too, totally not a waste to use what you love!

Reply on news.ycombinator.com