Review: A Dive into Mikrotik's Weird SmartNIC (2022) | Hacker News Reader

Review: A Dive into Mikrotik's Weird SmartNIC (2022)

alyx.sh

145 pointstodsacerdoti44 comments

She has an addendum here https://alyx.sh/posts/deploying-the-ccr2004-1g-2xs-pcie/

I just bought this the other day to setup a VPN to guard the IPMI interface in a 2U server I could cheaply house in some otherwise unmanaged rackspace. It's a pretty nifty piece of kit, but I still have to get better at understanding RouterOS.

I've been playing around with RouterOS, but it's not the easiest thing to get started with. There's plenty of documentation, but it often feels like they really want you to take the Mikrotik training, rather than you figuring it out for yourself. It's really interesting hardware though.

Does it support DPDK for fast packet processing or maybe SR-IOV ?. Given the price tag, I suppose it doesn't, but the documentation is sparse.

I dont think you can install your own OS on this, yet? :) , in order to do 25GBPS on ARM, I am sure they are doing some sort of DPDK ( I mean user space networking.). I would love to learn more on how do they achieve 25GBPS on this little arm64.

Almost guarantee that they aren't doing 25GBPS on the arm chip but on the dedicated ASIC.

I know for most of the Mikrotik platforms what gets offloaded to the ASIC and what gets handled by the CPU is a big deal in performance tuning and selecting the correct product for your use case.

I also think they only achieve about 15GBPS on their backplanes of their routers with SFP28, enough to show a benefit over SFP+ but not really full speed.

Lack of SR-IOV is mentioned in the article.

Had a router by them for a long time. It was fantastic but it took me months to set it up in a way that I'd be confident to put it to live service. It was a nice hobby project, and it paid off.

Now I had replaced it with a FritzBox as it's popular here in Germany, and my SO may be able to do something with it when I'm not there. As a new hobby project, I have a RB5009UG+S+IN lying around...waiting for me to get into the flow again...maybe around Christmas...

I love RouterOS and have been using it extensively for home and soho. And I absolutely agree: it can bee a pain to set it up. Every little part of it is complicated enough to spend hours on. Setting up networks can take long long hours. But documentation has improved lately and you can also watch their own YouTube channel, which has some useful videos. Also you should try using their forum, which is full of helpful guys. Don´t let yourself get scared, if someone gets very rough with you, though! On the other hand you can tune it in endless ways.

RouterOS really isn't any more difficult to learn than competing NOS like JunOS or IOS-XR, both of which also suffer from much in the way of easy-to-use Getting Started kind of docs for newcomers to high-end networking gear. Having used Juniper gear in my homelab for multiple years before I ever bought a Mikrotik switch our router, the hardest part was figuring out where everything was compared to JunOS.

I'm interested to see how many people saying that RouterOS is difficult to pick up are completely new to managed networking equipment, compared to how many are moving from competing solutions.

I think the crucial flaw here for the IPMI/BMC access use case is the fact that the card requires the server to be powered on to function. So if you accidentally turn the server off it’s game over for remote access.

Yes, but not a very deep problem in my case since I can order Remote Hands for this very niche scenario. A reboot doesn't let it loose power and that's all this is going to see for the forseeable future.

Most servers can be configured to always power on after a power failure.

Then all you need is a remotely-operated power outlet, which is pretty standard for colos nowadays. Toggling the power outlet is as good as a human driving out there and poking the stupid ACPI power-on button.

The only odd thing is the price:

Given it is the cheapest NIC in its class it should be a mass market product in high demand. The fact you get a free router built in is a cherry on top for the few who need it. If so why is it in such limited supply?

Alternatively it is a way to save rack space for people who need to squeeze in a router that doesn't fit. But if this is a niche marker router why is it so cheep?

> why is it so cheap

Using a high volume Amazon chip. Sadly the software stack is not what this thing deserves. It's a full blown DPU used with something very minimalistic SW wise.

> why is it in such limited supply?

The article makes a pretty good case that it's because it's poorly described, marketed, documented, etc. It might be a unique gem, but nobody can figure out what it can really do, what it's sweet spot is, etc. So, low demand means low supply.

It never really got a chance to be in demand though. It was announced, received a ton of press, had a delay u til a paper launch, then has staid somewhere between "hard to find" and "not available" since. A lot of MikroTik products tend to be niche but they aren't usually this difficult.

Maybe they're only given a limited supply of basically overstock chips from AWS. When AWS is building out data centers they need all the chips. Just speculating.

(Annapurna is the CPU company Amazon bought)

It seems that Jeff already has this SmartNIC tested with CM4 but not the latest RPi 5 new PCIe [1].

RPi 5 set up will make a poor man's high speed multi-port router with in-network computing capability, nice!

[1] Raspberry Pi PCIe Database: MikroTik CCR2004-1G-2XS-PCIe:

https://pipci.jeffgeerling.com/cards_network/mikrotik-ccr200...

While a cool experiment. I don't understand why you'd use this on a RPi unless your poor-man's server was already the RPi. Isn't that the beauty of this device is it's a router/firewall on a board that takes up 0-U space?

Because Broadcom pays people to spam internet forums with their Raspberry Pi brand.

Incessantly.

Normally I don't reply for self-evident answers but the sad sibling's comments make me reply anyway.

RPi is not originally made for industry usages but there are currently many industry that use it for prototyping and even inside real world implementations due to the existing eco-systems for examples drivers, software, and hardware (daughterboards, hats, etc).

In my case I have designed Linux based network training kit with open source software like Quagga and LiSA (Linux switching appliance). It always good to use widely supported hardware like Rasberry Pi, and this Mikrotik NIC is good for data forwarding plane and network acceleration with potential extras like in-network computing and edge processing similar to SmartNIC and Data Processing Unit (DPU).

Apart from training we also have collaborations with China and Australia Radio Astronomy groups and one of the interesting part is to perform e-VLBI. It is nice to have a device in one small appliance (RPi + accelerator) in the remote observatories that cost less than USD300 with much lower power consumption rather than a full blown Xeon based server with accelerator.

Looks like these are now in stock at Getic for $160 each.

https://www.getic.com/product/mikrotik-ccr2004-1g-2xs-pcie

I've bought Mikrotik gear from them before. They ship to the USA.

This is where I got mine, came in last week. Zero issues.

Just got one of these in for my home lab. Haven’t tossed it in to play with it yet, but this is one of the few reviews out there for it. Fits my use case perfectly.

>suggest using it in a file server and a workstation, with a copper DAC in between

What is dac in this context? I'm guessing it isn't digital analogue converter.

Direct Attach Cable, ie an end-to-end copper cable.

https://www.servethehome.com/what-is-a-direct-attach-copper-...

Direct attached cables - vs. an optics module with fiber that you then plug into it, or an rj45 module you plug a cat 5/6/7 cable into.

Ok, my only other guess was digital audio cable.

TLAs (three letter acronyms) are hard.

In the network world context it's quite obvious and I guess that is what it is written for. For those not in that context I guess it's quite understandably confusing.

Also either network people get it wrong, or the optics modules really are DACs? Now I am confused too haha : https://serverlabs.blogspot.com/2019/10/a-study-about-sfp-mo...

Tangentially… you could build a SFP module that uses a digital to analog converter, outputting the data as a range of voltages vs 1/0s. Hook it up to a speaker for terrible audio. Then listen to it with a microphone hooked up to an Analog to digital converter sfp module on the other end!

99% of the time when someone is talking about 'DAC' in a physical networking context it will be direct attached cable.

These types of cables are literally just copper wire connecting pins at either end. In comparison to fibre optics which require transceivers with their own logic and encoding circuitry within the SFP modules themselves.

Edit: I think this article is actually pretty harmful, I don't think people would normally think of networking as DAC/ADC...

> When it comes to copper cable communications, digital signals are converted as electric signals (analog signals).

The signal is still digital, it's not "analog" at all. Is data being read from your hard drive over a SATA cable "analog"?

You would call the chip that does the encoding for the physical medium the 'PHY' chip - not an DAC/ADC.

Fully agree. Thinking of network transceivers as digital to analog converters (DACs) is just as conflating as thinking of pulse width modulation (PWM) controllers as DACs. Analog to digital converters (ADCs) and DACs are referring to the fact the actual analog signal you input/output is the ground truth value of the data not that the digitally encoded signal is being sent in the real world on a wire. The difference is if the signal is distorted on the way to the other end of the cable it's still (within tolerances) decoded back to the same digital value despite the changes in the physical signal.

Yeah I totally agree with this as well.

Link to the product site which also includes an embedded video: https://mikrotik.com/product/ccr2004_1g_2xs_pcie

Today I upgraded RouterOS from 7.10 to 7.12 on a hAP ac3, which went well, and on a CRS, which wiped the entire config, and the cloud backup was not restorable. Trying a subsequent downgrade (so I could perhaps restore the 7.10 cloud backup) soft-bricked the whole thing and I had to fix it with NetInstall. Maybe I will have to start exporting the config to a fucking text file, I guess.

It's got an AL32400. At least my exemplar, I have removed the heatsink once. Also you normally don´t have to worry about the lifecycle of a Mikrotik product. Almost everything they released since they were founded is still usable and you can install a currently supported RouterOS version on it.

One more addition, as I've just seen the addendum of the review mentions that there is almost nothing in the user manual: the Manuals in the Mikrotik world only describe HW specific quirks. You have to go to: help.mikrotik.com Then study it for a few hours, days...

Here is the description for passthrough: https://help.mikrotik.com/docs/display/ROS/Ethernet#:~:text=...

I'd love a device like this that was simply a three-port gigabit switch (not a router).

With most colos if you ask for a second network drop they make you pay a fee every month, forever. This way I could pay for my custom BMC's network connection once.

You can bridge the ports, if you want. But you would probably want to hide BMC access behind a VPN.

This Mikrotik device has a third port, a 1000Base-T one. That should be plenty for BMC.

I have a custom BMC precisely so that I don't have to hide it behind anything.

I am thinking of using this. But it seems like overkill, and I am not interested in learning RouterOS.

It is definitely overkill.

RouterOS can be managed using a GUI, with an app called Winbox. It is windows application, but runs fine under wine, arm macs included. You still need to know the concepts, but in a nice clicky way.

I love the idea of a firewall on the NIC.

The more defense-in-depth, the better.

Reply on news.ycombinator.com