Hacker News Reader

I also never thought I'd bother with 10GbE until I stumbled on some used Mellanox ConnectX-3 cards that were just cheap enough to play around with (~$30/card including DAC cables). Seeing file transfer speeds of >700MBps between my server and NAS convinced me it was worth keeping them installed, though initially just as a point-to-point link between those two systems.[1]

A few months later, I was upgrading to a 24-port managed switch, and found one with four SPF+ ports (Aruba 1930) for little more than I could find a purely gigabit switch (~£180). I moved the 10Gbps NAS/server link to the main network, and added 10GbE to my main PC (with another ConnectX-3), and it's been a really nice system to use. A little overkill for most of my usage, but it's nice to have.

I'm not convinced it's worth the cost of moving most of my other devices to 10GbE, even though I have a >1Gbps (down) Internet connection, as switches with more SPF+ ports or any 10G-BaseT ports (which some of my devices would require IIRC) seem rather expensive. Like you, I'm also open to suggestions if there's a cheap(ish) option I've overlooked.

[1] Side note: I actually moved a USB drive from the server to the NAS as it had significantly faster (by about 30%) sustained read/write speeds that way. I'm presuming the NAS is caching the requests in memory, but both are on a UPS and it's only temporary working data anyway so I'm not too worried if the data doesn't actually hit the disc immediately.

For about $50 US, you can get a 10Gbase-T SFP+ module that gives you a copper port at 10gig within a SFP+ form factor. That, coupled with a cheap Mikrotik switch, is plenty to get started.

Second hand Brocade ICX switches are also plentiful and not too power hungry (but they can be loud).

Just a heads up to anyone who does intend to go buy a bunch of those modules.

They run absurdly hot, and they eat a lot of power.

If you don't have very good ventilation then you may find these modules running at well north of 80C. I ended up having to mount a 120MM PC fan and attach some mini-headsinks to them to keep them under 50C. Just sitting on a desk they were too hot to touch.

You may find that the power budget for your small 8-12 port SFP+ Switch isn't enough to run more than a few of these at once. I have an 8 Port Mikrotik SFP+ switch that can only run about 4 of them. The Quad-Port SFP+ NIC I bought from fs.com (Intel XL710-BM1) can only do two.

Expect to also run into issues if you use old, poor quality or mistreated cables. If you bend them too far (trying to get around a sharp corner) - they will work fine at 1Gbit, but just not work at all at 10Gb.

You may find it's a whole lot cheaper (and more reliable) to just get some optical cable and transceivers if you want to go more than a few meters.

For distances less than a few meters DAC cables are often reasonable value.

From personal experience, I'm now leaving the 10G-BaseT transceivers for runs where I can't run optical (like existing in-wall wiring).

Yeah, optics or DAC cables are definitely preferable. I wouldn’t run more than one or two 10G-BaseT units in most switches - but when you have a modem or device that only supports it, it’s a much cheaper way to get connected than buying a dedicated switch.

Woah, the brocade switches might be a very good option. I don't mind it being loud, since my server rack is in a well insulated part of my basement and have had no noise issues even with a couple 2U servers. Thanks for the suggestion!

Depending on how many ports you need, you may get away with an ICX 7150-12. It has 14 fixed Gb copper ports and 2 10 Gb SFP ports. It's fanless and has 12 GbE PoE ports.

Do you need to have 10GbE? That is very high end kit, especially with copper. The power consumption is extreme and the distances are never going to be very good compared to fiber.

2.5GbE ethernet is super accessible now from Netgear, Trendnet, TPLink etc and doesn't require new NICs on any newer desktops. All my shit that 2.5GbE is useful supports 2.5GbE.

I'm not sure actually, I have a local NAS/Server rack where I store work related datasets (think a few hundred/thousands 12-15GB .h5 files). And while I definitely don't need instant or even super fast access to them, I'm sure it wouldn't hurt either. We already have tons of optimizations and usually process datasets in a cloud environment with "good" IOPS/Network[1] speeds, but it is still faster to use locally in some situations. 10Gbe gear would be a nice to have to avoid bottlenecks within the LAN... but I just realized that I'm making a good case for just shelling out more money instead of cheaping out haha.

[1] As good as it gets on Azure, I guess. Which is not much at all, even on Ultra SSDs so I keep a local cache for smaller scale stuff

Since you only have 1.5gbs, why don't you just get a 2.5g switch? Those are much easier to find and can work with plain cat 6 Ethernet

I guess to me 2.5gbe just seems like a weird in between with less support, but I agree that it's probably going to be my only option otherwise. Though a 10Gbe switch would also be awesome for local lan, since my servers' NVME SSDs are constantly "throttled" by my 1Gbe (and would be at 2.5Gbe too, but to a lesser extent).

10Gbe is really best with DAC or Optical cables.

Even for really short runs? I'd mostly use it for the link between my ISP modem to the switch, and wouldn't mind slower speeds for the rest of my devices. But that's definitely something I'll have to keep in mind.

TwinAX has lower latency and power requirements than fiber.

https://www.arista.com/assets/data/pdf/Copper-Faster-Than-Fi...

10GBASE-T is the worst within-building 10Gbit networking choice, unless one can't use fiber when the distance exceeds the baseband twinax copper DACs.

> […] almost too-good-to-be-true symmetric 1.5gbs FTTH with Bell Fibe.

One issue I have with Bell Fibe is the new HomeHub 4000 hardware is mandatory, and the GPON optic is not removable (the one in the HH 3000 is).

I'm not exactly enamoured at having to use Bell's hardware (though I'm sure it gets the job done for most people).

I think you can still get the HH3000 if you manage to escalate a request past the first layers of consumer support.

I also heard about buying special optics that you can completely set up to mimic the original bell GPON, down to the SN but those are expensive. It's so weird to "downgrade" such a useful feature (that does not even hurt any other part of their business, and only serve to make 1.5gbps+ speeds more attractive for those who would actually need that much speed), but hey it's still bell after all