The Cheap 10GbE Switch Buyers Guide | Hacker News Reader

The Cheap 10GbE Switch Buyers Guide

servethehome.com

52 pointstranspute20 comments

Quick note on the CRS312: while absolutely lovely as a switch, you have to be careful about what exactly you want to do with it, as, as with the entire Mikrotik range, it can do absolutely everything (provided you can wrap your head around the configuration), but isn't necessarily good at it.

In the case of the CRS312, its potato-class CPU means that even NAT at 10Gb/s can be challenging. However, if you pair it with something like a CCR1036 (discontinued, but still available and sporting a 36-core CPU) or its successor, the CCR2004 (or even something like the RB5009UPr, which adds PoE, but is limited to 2.5Gb/s) you'll be all set.

I think this is good to point out, but I feel like you left out the reason why this is so. Mikrotik allows installing of both SwitchOS and RouterOS. The CRS is a switch. That means it can do vlan-aware switching between all its ports, and it’s a managed switch. It has ASICs that can do line rate switching of 10gbe on all ports simultaneously. You can use it as a router (NAT is something a router would typically be responsible for) but that will be super slow because it’s not using the switch hardware. The CCR2004 is a totally different type of product. It’s a router, not a switch. It can do NAT faster because it has a faster CPU, but it also can’t switch at line rate across all its ports simultaneously because it doesn’t have the ASIC switching hardware. Mikrotik is good about providing both the switching and routing performance metrics, but it’s kind of unclear unless you know what you’re looking for.

It's not a SwitchOS (which can and should be ignored) versus RouterOS issue, more that Mikrotik is limited by the capabilities of the off-the-shelf switching SoCs they have to use due to their size.

Once you run into limitations there (which can already be due to something like "needing to use VLANs"), it's all CPU-based, which isn't necessarily a problem (the amount of switching performance you get out of a Tile-based 'router' is pretty amazing).

The most interesting piece of documentation about any piece of Mikrotik kit is the block diagram, followed by close reading of the L2/L3 offload docs...

Might be a misconfiguration, but according to https://help.mikrotik.com/docs/display/ROS/L3+Hardware+Offlo..., the CRS312 has hardware NAT and Fasttrack. That should allow 10Gbps NAT as long as the number of connections tracked are below the hardware limit. Anything beyond the limit will be managed by the CPU.

The CRS312 only supports 2K NAT entries in hardware, which is very little.

Yes, it does have hardware offload, but that works better in theory than it does in practice. (For doing line-rate NAT with more than a handful of clients, it's not an option, mostly because of the state table limits already mentioned in a sibling comment).

"Will the hardware offload actually do anything useful?" is a time-honored Mikrotik game: see the sometimes surprising limits at https://help.mikrotik.com/docs/display/ROS/L3+Hardware+Offlo...

True but I doubt someone serious about networking is going to be confused about this. The CPU in a switch is just for config and management. MT is flexible in that you can do some routing with the switch CPU but that doesn't mean you should.

I've seen plenty of people getting tripped up by this, sometimes quite publicly (like the blogger who returned the Mikrotik switch they got for their 25Gb/s home internet connection because turning on NAT turned it into a noisy space heater...)

That 36 core number is all marketing. Can only do so much with 40nm at ~25W TDP.

Marketing or not, it's a useful metric when comparing expected performance across the Mikrotik product line...

Why are you talking about NAT in a thread about switches?

What matters to me, as a home user, looking for some affordable high-speed networking is; can I do 10GbE line-speed switching over VLANs on something like the 305 or 309?

Can I do 2.5GbE to something like the Unifi WiFi 7 AP, again with VLAN trunking?

I have a router for routing (Protecti box running OPNSense) and besides faster access between a high-speed AP and my storage servers, I might also just want to connect my desktop and NAS, as well as have a high-speed link over copper say to my living room where I might want to put a 1GbE switch with 10GbE uplink for the various devices beneath the TV to connect back to the storage servers etc.

I want to do this with as few additional devices (preferably no more than a single 10GbE switch in my network cabinet, a NIC in each device and one more switch for the living room; I already have a bunch of 1GbE switching in a mix of PoE and non-PoE in Ubiquiti EdgeMAX equipment), and as little power use as possible.

I just love my eBay Juniper gear. There's a fan less PoE 12 port switch which is line rate forwarding, vlans, managed, etc, the Juniper EX-2200-C (compact, fanless very large heatsink industrial design). There's fanless Juniper SRX firewalls that'll do 1Gb/s NAT and L3 just fine. Yah it's an old proprietary freebsd binary blob network OS to commercial silocon, but they're solid and do REST and custom controllers if that's your thing (can build overlays and calico kubernetes with bgp on em even)

ok, but no 10Gbit

I am a big fan of the Arista switches you can find on eBay. Excellent support for RDMA/RoCE

Is it easy to obtain up to date firmware updates for old switches?

Their used 100gig on ebay are surprisingly "affordable".

I like old Brocade switches. Really cheap.

Last month there was a good supply of used Mellanox SN2010s, a 16x25G+4x100G x86 white box switch/router capable of being a firewall at price of ~1000$. But it's all gone now.

Whats the compelling reason to buy any of these vs picking up another arista 7050tx for sub 300 bucks on ebay? Atleast for a home user that can put the noisy switch somewhere the noise isnt a big deal,i dont see a lot compelling here vs the 2017 vintage gear that was 20k new and is now being offloaded with lots of life left

Can any of these do 10Gb on all ports simultaneously?

Reply on news.ycombinator.com