Hacker News Reader

25Gb NICs are getting into the affordable range, but SFP28 modules are still very pricey in comparison. I’m a big fan of the Mellanox CX4 LX cards for their low power draw (11w max). OEM cards come cheap and can be crossflashed to generic Mellanox firmware.

The bigger issue with 25G is that it’s well into the range where any OS’s default TCP settings won’t provide anywhere near line speed, and once you solve that it can expose other non-network bottlenecks. I have dual-SFP28 cards in both my workstation and NAS, both connected to the network via a10G Mikrotik switch and directly to each other via a 25G point-to-point link. Now after all that tuning I have a fast network but run into the SAS bus bottleneck for any file transfers exceeding the size of my app take write cache :p

> SFP28 modules are still very pricey

SR is ~$40 and LR is ~$60, so more expensive than SFP+ modules but not out of the reach of someone who even has a capable SFP28 port.

25GB/s is overkill, and 640K ought to be enough for anyone. Those quotes age well.

1 Gbit/s is pretty close to still being overkill. 25 Gbit/s is laughable overkill and appears very likely to remain that way for the coming decade.

Consumers around the globe have had increasingly common access to 1 Gbit/s for a decade and there still aren't any other great, common use cases for it beyond very high quality video streaming.

It didn't take very long for computer use to need more than 640K by comparison. In the computer realm those edges were being constantly pushed at that time. Such is not the case with bleeding edge consumer broadband speeds today.

1 Gbps is slow. Even recent wifi can plausibly exceed 1 gpbs to a client. Pretty much any modern HDD (let alone ssd) can read or write faster. USB 3.1 is an order of magnitude faster and display port 2.0 tops out at about an order of magnitude faster than that. It doesn't really make sense to leave the main physical network interface so far in the dust, let alone claim it's overkill.

1 Gbps is laughably slow. It doesn't even keep up with hard disks. Network attached storage is crippled by 1 Gbps networking. It's ancient. I remember doing an assignment in 2005 to design a network on a budget, deciding to "splurge" on gigabit, and finding it very much affordable. That was 17 years ago, and yet consumer networking barely budged since then.

10 Gbps is still below the 7 GB/s that a single NVMe on PCIe 4 (which is readily available) can achieve.

25 Gbps is still below that.

I'd say 100 Gbps is where the current practical maximum is more or less. You'd have a hard time writing or uploading data that fast on anything resembling consumer hardware.

you mean 640GB of RAM ought to be enough for anyone. 640k wasn't cutting edge at the time, it was meant for personal computers.

When he was asked about that Gates said he never told that, because never ever nobody could assume what $somenumber is enough memory for anything.

"Everybody believes quotes on Internet" - Abrahamo Lincolni

definitely overkill for now. But looks like author had fun setting everything up so why not?

The init7 CEO said they will offer 100gbit symmetrical in 2-3 years as the SFP modules become more mainstream.

640K Ought to be enough for anyone ;)

Granted, it's beyond the definition of overkill (but I wouldn't mind it!)

Wired, you can do 10Gb/s for server systems, which are, amongst other things very loud and not very suitable for placement anywhere near humans.

My 10gbaset switch is all of 23 dB when the fan kicks on and I have other switches with sfp+ that don't even have a fan.

In what world is 10GbE only available for servers?

Without drowning in fan noise? This world... Sure, I guess you can get a Mac Studio, or some other 'workstation' class PC, but your switch will still need to be within a few meters of that endpoint, and it's not going to be very green nor silent.

2.5Gb/s can easily be done with a lot of laptops and workstations these days; 10 Gb/s isn't quite there yet (and 25, 40 and 100Gb/s are definitely in the server-only fiber-or-DAC-only realm)

Mikrotik makes a whole bunch of 10G hardware. The 8 port ones are fanless and have a heatsink on the back, and the 16 port model has a fan that only gets switched on when needed. I have one sitting behind me and it's been off all day.

It's also easy to open and replace the fan with something less noisy. Just remove a few screws, it's a standard size with a normal connector on it.

There's also big external heatsink so you could rig up a big, slow, fan to help it out a bit.

If you want to be green by the way, use fiber/DAC. The 10G copper SFP+ modules are power hungry, and Mikrotik recommends not placing them next to each other. Also the extra power draw is likely to result in fan use if you have a lot of them.

I had issues with 10G copper SFP+ modules even when there was only one installed in the switch (Mikrotik CRS305-1G-4S+IN), the other modules installed being DAC/fiber. I got random disconnects I could not attribute until I checked the switch logs - the module was shutting down because it was reaching >90C when the ambient temp was 26C. I had to add a fan.

Not all transceivers are created equal; some need more power (and then dissipate the heat), some are satisfied with less. Some can do only 30m distance, others will run over 80m distance.

The Mikrotik ones (S+RJ10) are based on Marvel chip and they are the more power hungry / run over 30m only variety. On the other hand, they can negotiate 2.5G or 5G if necessary and support a proprietary protocol to tell the switch about it, so you will that in SFP+ properties.

As I have written in the sibling comment, I have good experience with BCM84891-based transceivers. CRS305 can handle two (still not next to each other, obviously).

I have the Mikrotik 8 port, but the spacing requirement kind of sucks. If you want RJ45, I'd go with this one: https://www.qnap.com/en-us/product/qsw-1208-8c

If you want RJ45, they also have a switch with 10G RJ45 ports built in: https://mikrotik.com/product/crs312_4c_8xg_rm

Oh awesome, and pretty well priced! Know what the noise level is like? The QNAP is designed to be in a home office rather than a server closet, so it’s inaudible. But my mikrotiks are passively cooled, so even better.

> but your switch will still need to be within a few meters of that endpoint, and it's not going to be very green nor silent.

Mikrotik CRS305 (4xSFP+) and CRS309 (8xSFP+) are both passively cooled. They are pretty much silent :) though the blue led takes some tape to be less shiny.

> 10 Gb/s isn't quite there yet

If you really, really need RJ45, look into BCM84891-based transceivers. They still get hot, but not as much as others (according to datasheet, takes 1.6W at 30m and 2W at 80m). I also managed to get stable 10g over 20m Cat5e with them.

If you have a PCI-e slot, there are reasonably priced, passively cooled NICs. Here's an RJ45 one: https://www.bhphotovideo.com/c/product/1344847-REG/asus_90ig...

There are a few quiet 10 GbE switches, the passive Mikrotik ones others are mentioning, but this one is a quiet actively cooled one if you want RJ45: https://www.qnap.com/en-us/product/qsw-1208-8c (Lots of combo RJ45/SFP+ ports)

>but I fail to see a use case over my 500Mb/s home connection.

The last time I thought I have not enough BW on my 50Mbps connection I learned what it's just my T440 is not enough to decode h265 extremely compressed stream over an SFTP streaming over WiFi, which gave around 20 Mbps at best. After toying|fighting around with wireless settings I found the main culprit was my Intel 7260 (or whatever) WNIC, not the CPU, Internet connection BW or the server throughput (10Gbit, despite being an IIS instance).

YMMV