Comment by sliken | Hacker News Reader

Comment by sliken | Hacker News Reader

Seems like MikroTik and Ubiquiti haven't been particularly good on documenting how their routers work, what each process does, and how to debug/verify what they are doing. I've been particularly worried about Ubiquiti, since they seem to be ignoring the MIPS based EdgeRouter line. The various consumer NAS vendors have been a nightmare security wise, and it's far from clear that the prosumer routers from Ubiquiti and MikroTik are any better.

Should it really require reverse engineering to figure out how a Linux box passes packets?

I gave up on Ubiquiti and bought a tiny $120 router ($140 with a nice metal case) that's a NanoPi R6s. Pretty impressive specifications. 8GB ram, 8 cores (4 fast Ax76 and 4 slow A55s) and no fans. Has 2x2.5GBe and 1x1GBe for networking. I've installed a port of OpenWRT called FriendlyElec and Ubuntu 22.04.1 LTS. I didn't bother cross compiling, it's plenty fast for native compiles.

I've been impressed so far. It compiles Rust about half as fast as my Quad core Xeon server from years ago, and is SEVEN times faster than a RPi 4 8GB! I have an Apple M1 mini around that manages the same compile in 13 minutes. A nice bonus is neither network or storage is USB attached, unlike the RPi.

Burned it in overnight, running all 8 cores flat out, with no problems. Haven't decided what to use, maybe shorewall or just raw nftables/iptables.

Similarly there's 4 and 6 port 2.5 Gbe boxes with various N5000 and N6000 celerons at around $200. ServerTheHome has reviewed many of them, they seem to be evolving nicely, better cases, better heat sinks, better 2.5Gbe Intel chips, etc. Run *WRT, pfsense, or whatever else floats your boat. STH even demod running the firewall under proxmox on one of these cheap 4-6 port 2.5GBe widgets. Would be nice to keep your complete config in git or the config management widget of your choice (often Puppet or Ansible).

Guess I'm just getting less trusting in my old age.

You've thrown out a lot of performance numbers here but most of us aren't compliling rust on our routers. Have you stood up iperf to see how fast it routes? The task is harder than you think if you want any features to go with that and there's a good reason most vendors (Mikrotik and Ubiquiti included) offload most of the task to silicon.

I suspect the majority of consumer gigabyte routers are never saturated. The technology has eclipsed typical demand. Not that there should be deceptive marketing or performance cheating, but I suspect 95%+ of people will never be limited by their network gear.

Yeah, sure, but it's frustrating. Sure casual users who won't saturate the network get fast offloaded performance.

However more aggressive users will end up enabling more complicated firewall rules, QoS, buffer bloat mitigation, or deep packet inspection and disabling the silicon based acceleration or bottlenecking elsewhere.

Thus my interest in the RK3588, that provides 4x the PCIe of the RPi4 and a fair bit more CPU.

I do wish there was a popular benchmark that used a more realistic benchmark. Smaller packets, dozens or 100s of TCP sessions, non-trivial firewall rules, etc. Doesn't seem like you learn much from iperf managing 95% of of line rate with large packets.

But here's the iperf numbers I stumbled across: https://www.cnx-software.com/2022/11/12/nanopi-r6s-review-un...

I think building it yourself works well for routers and NASes, but there's a gap for WiFi. Lot of the bells and whistles that make modern wifi perform well (beamforming, MIMO, ...) is locked behind proprietary software and your homemade access point will never work all that well.

Interestingly, this is also something MikroTik isn't very good at - their long-range wifi is nice, but their consumer wifi used to be stuck at 2x2 and basically worse than cheap Asus "wifi router" which is somewhat embarrassing.

For switches, if you want fast, for small installation it is hard to beat the Mikrotik 4xSFP+ 10G switch for $150.

you are drinking too much of their marketing.

Not a single wifi consumer vendor have any say on those features. it is one hundred percent done on the closed OS in the radio chip.

your expensive ubiquiti/oanda/cisco have the same mimo/beanforming performance as any opensource/clone using the same wifi radio chipset.

the consumer facing OS just flip a bit somewhere in the configuration flash. granted, knowledge of the right bit to flip mighty be missing on the opensource still, but there's no magic happening in ubiquity owned code.

uniquity is extra shaddy as they buy off the shelf components but demand custom labeling to look like it's their custom silicon. it's not. uniquity is the matress store of wifi.

> uniquity is extra shaddy as they buy off the shelf components but demand custom labeling to look like it's their custom silicon. it's not. uniquity is the matress store of wifi.

They're not - I have no idea about the hardware itself, but what makes Ubiquiti so popular is the software integration layer. Their stuff Just Works even for incredibly complex and large installations while still being easy to configure.

I've also replaced a Ubiquiti EdgeRouter with a spare PC I had around. It was a small form factor thing, I don't need WiFi, so I replaced the WiFi card with a M.2 Ethernet card[1], which gives me two network interfaces. The point is pretty much anything is powerful enough to be a router now.

(Actually my setup is more complex, as the machine is powerful enough to run a few VMs, so I pass through one of the PCI devices to a firewall VM, which means the VM host doesn't even see the outside world.)

One thing to watch out for is Intel released some bad 2.5Gbe hardware (e.g. earlier versions of the I225-V). I even have a more recent one that seems to have issues (even when running the latest Linux from kernel.org). You can find Realtek cards though, which seem to work better (yeah, who'd have thought that would be the case).

[1]: It was something like: https://www.amazon.com/Suroene-Ethernet-Adapter-Multi-Gigabi... -- there are others.

Apparently I225 went through 2-3 revisions, and the "b3" is supposedly ok. But the newer $200 PCs are now coming with the I226, which it's claimed is much better.

Re that M.2 Ethernet card, that's a long untwisted ribbon cable. 100Mbit should be fine, but 1G or 2.5G? Sus.

I take it you've seen PCIe riser cables like this one:

https://www.amazon.com/Zer-one-7-5Inch-Extension-Flexible/dp...

If that sort of thing can work at PCIe rates I don't see any reason why that M.2 card can't. It's a lower bit rate plus theoretically a more resilient encoding for long distance as opposed to PCIe.

The untwisted ribbon cable is fine for PCIe in short runs by virtue of PCIe data lines being a differential signal and if the length of cable is kept short(-ish). The max length of the cable should vary inversely with the generation, that is, max length of a flexible copper extender for gen v1 is longer than gen v3, more length introduces more noise which increases error rates. A well terminated cable should be able to use the full bandwidth of the bus. I'd be completely comfortable running x16 flat out at max rate with one of those.

I don't have any hard-and-fast sources to back the above up, but punching 'pcie extender max length' into the Gargler should find you something to suffice.

There also are dual 2.5G Ethernet (and multiple SATA) adapters for M.2 sockets, which make things even more interesting for those little ARM boards.

https://www.aliexpress.us/item/3256804337466480.html

https://www.aliexpress.us/item/3256804400925663.html

OpnSense/XigmaNAS devs, are you listening?

I bought one of the Edgerouters thinking it would tie into their Unifi Web controller interface - no hells bells - got a Mikrotik like device where you need Cisco IOS level of skill to configure.

Got a TPLINK Omada ER605 router now - just as powerful but way easier to configure.

> got a Mikrotik like device where you need Cisco IOS level of skill to configure

That's giving it too much credit. I have experience with Cisco IOS and NexusOS, and I'll say that Mikrotik's RouterOS was hard.

The funny thing is that when I was big on ubiquiti I always went the other way edgerouter only. mainly because I did not want to get tied into the whole unifi ecosystem.

They sold two versons of their switches and routers, the black ones(good) which were stand alone and the white ones(bad) which depended on the unifi system.

But then again I am also the idiot who ran openbsd on my edgerouters.

> got a Mikrotik like device where you need Cisco IOS level of skill to configure

You have not used a Mikrotik wifi router in the last 10 years.

If you're considering straight nftables, do straight nftables. The syntax isn't bad for editing/reading directly. The userspace tool has quirks but you'll get the hang of it. I run my config through a templating engine (the same ruleset goes to cloud VMs etc).

My main router is virtualized on a Ryzen 5600G that also does other tasks. I have two other wifi APs that are just low power x86 boxes that also run Kodi.

Getting back to the original topic, I've still got Mikrotik for switches. The hardware is decent, the software is janky. I've got three different flavors of config - SwOS, RouterOS, and old RouterOS that requires a different type of config because the older switch chips were never updated to the newer config commands.

Mikrotik continues to hold their software too tightly instead of making the transition to a hardware company. They've built so much routing functionality, that's just pointless on devices that are switches. Yet there is some switch functionality that isn't available on SwOS. And SwOS is web-only with a binary config backup, rather than something conducive to centralized administration.

I'd love to find some low power white box 1Gb+10Gb switches that could run straight open Linux, or even using my existing Mikrotik switches that way if someone were to forcibly open their the environment.

> Would be nice to keep your complete config in git or the config management widget of your choice (often Puppet or Ansible).

I did this with my (mostly) OpenWrt-based network: https://github.com/kwesthaus/network-config.

I chose the NanoPi R4S instead of the R6S as my router since it is fully supported by vanilla OpenWrt [1] whereas the R6S is still in the works [2].

[1]: https://openwrt.org/toh/friendlyarm/nanopi_r4s_v1

[2]: https://forum.openwrt.org/t/nanopi-r6s-a-rockchip-rk3588s-ro...

I saw an Armbian release for a similar RK3588 sbc, hopefully that image works and had a mainline kernel.

Don't know about the Linux distros, but one problem with the FreeBSD versions (pfSense, OPNsense) is the mostly nonexistent WiFi support because there's no driver support for PCIe cards. You end up needing to hook up a COTS Wifi AP, which has all the same problems that a COTS router would've had.

But other than that, yeah I don't see any reason to use a COTS router when I can just make one. My current router is my old desktop PC that became redundant when I built a new one, sitting as a bare motherboard on a test bench to take up very little space, running OPNsense.

How does a common wireless AP have the same issues as a common consumer NAT router? Do APs have to worry about NAT state tables? Are they exposed to the Internet? Do they have to be configured to NAT/forward special edge case devices, like Nintendo / Sony games consoles?

Complaining about the lack of wifi drivers is pretty silly if you don't understand the advantages of a simple bridging wireless AP.

>Are they exposed to the Internet?

They're exposed to WiFi. They have terrible security in baked-in firmware that can't be updated without connecting them to vendor clouds, if at all.

FragAttack was just last year.

>Complaining about the lack of wifi drivers is pretty silly if you don't understand the advantages of a simple bridging wireless AP.

Ah yes, because one can think of three things that APs are not affected by, therefore APs are bullet-proof.

Cloud-based APs? Seriously? Nobody who actually takes security seriously would use cloud-based APs. Even using cloud services to download firmware updates would be shady as heck.

You just made my point for me. If you have to download binary blobs to your hardware, like wifi interfaces, you can have issues. Would you rather have vendor-dependent devices inside your firewall / NAT router, or just connected to the same network as your firewall / NAT router?

Nobody said APs are bulletproof, silly. Only people who don't know better would suggest or even imply that having separate APs is somehow worse. You want to keep the less secure stuff over which you have less control out of the important systems.

Are you using the images available from FriendlyElec? I have limited trust in them.... Been struggling to find a good image except for DietPi.

Heh, it came with FriendlyElec, I just used it long enough to make sure the hardware was working and stable. I always reinstall Linux from something trusthworthy upon receipt.

However all the NanoPi images are pinned to the same 5.10.x kernel, that I don't think it upstreamed, which is pretty ugly.

Armbian just had a new release, supporting a TON of SBCs, even a Risc-V board, which is weird based on the name. In any case, they support another RK3588 board, so I was going to try that. I want something using a mainline linux kernel if at all possible.

> I've installed a port of OpenWRT called FriendlyElec and Ubuntu 22.04.1 LTS.

Which one did you decide to stick with? Or are you using both?

The problem I've had with OpenWRT is updates are a pain.

OpenWrt upgrades can be made MUCH more convenient by installing extra packages that enable "Attended Sysupgrade" - check https://openwrt.org/docs/guide-user/installation/attended.sy... for more info, and please spread the word if you like it! :)

The 1x1GBe port on your system is the controller built in to the SoC. It supports jumbo frames so you can do full size packets over PPPoE if your ISP can handle that.

Reply on news.ycombinator.com