OPNsense DEC740 Ryzen Embedded Fanless Firewall Review (2021)
wiki.junicast.de
wiki.junicast.de
$200 used: The venerable PC Engines APU2 is a fanless x86 AMD 10W TDP router with 4GB ECC RAM, TPM 2.0 and GPIO pins, open schematics and coreboot, which can run pfSense, OPNsense, OpenBSD, Linux, FreeBSD and OpenWRT, with virtualization support. mPCIe slots for WiFi, LTE & mSATA. Constrained by supply chain at present.
$200 used: HP t730 and t620 Plus thin clients have an AMD SoC with similar TDP to PC Engines APU2.
$200+: MIPS-based Ubiquiti EdgeRouter Lite/4/6/8 can run Linux and OpenBSD (octeon/MIPS), but is also supply chain constrained.
$200+: Intel-based https://protectli.com/ (coreboot) and virtualization-capable multi-NIC mini PCs s with unknown UEFI, https://news.ycombinator.com/item?id=31451142
$400 AMD/Xilinx wildcard dev board for Robotics, KR260 has Arm SoC, Xilinx FGPA and multiple 1GbE ports + 10GbE SFP, backordered 20 weeks, https://www.servethehome.com/amd-xilinx-kria-kr260-robotics-...
$400: ASRock 4x4 dual-NIC has Ryzen Embedded or 4000-series SoC with Realtek NICs, questionable BIOS and limited support focus on Linux/BSD.
$400 used: HP t740 thin client with Ryzen Embedded is Mac Mini size, with PCIe slot for low-profile quad-port or SFP NIC, https://www.servethehome.com/hp-t740-thin-client-review-tiny....
Some SOHO networking devices made by Microtik, QNAP and Ubiquiti contain Arm SoCs made by Amazon's Annapurna Labs, who also make AWS Nitro silicon, https://en.wikipedia.org/wiki/Annapurna_Labs
You might as well just buy a Dell SFF with a low-powered i5 at that point.
Ryzen Embedded SoC includes 10GbE networking, but until this Decisio device, I've not seen any OEM using it. Maybe there was a chicken-and-egg issue with OS drivers.
At the high end, Microsoft/LF SONiC + OCP merchant switch silicon has provided an open networking platform for 25GbE+ data center networking.
So I have a few EdgeRouter Lite 3s doing NAT and port forwarding (IPv4) because they can do that in hardware. As they are not open and I don't really trust them, I just pretend that they are on the ISP side of my network (even though they are mine, and I "control" them). So their LAN ports count as the "outside" in my setup. They provide networks like 192.0.2.0/24 that I just pretend are my public IP's.
This is also where the AP with GF's phone and chrome-cast resides on a separate VLAN. This has the benefit of me being able to play around with "real network" behind the firewalls in the next layer without worrying about causing downtime and issues for her (big win).
After that I have my "real" firewalls (a bunch of APU4C4s) that segment my network into several parts and multiple layers. They have less work to do as they don't need to do any NAT/translation. It also makes the firewall rules much simpler as there are no NAT/RDR rules and I don't have to think about whether a firewall rule apply pre- or post- translation (NAT/RDR).
It would of-cause be a lot easier if I just had a /24 of public IPv4 addresses, but this setup lets me sort of pretend that I do, even though I only have two static IPv4 addresses and an extra one with DHCP.
I just recently got IPv6 with a /56 routed to me from my ISP and this is what I am messing around with currently. I have some Juniper switches (EX2200/EX3300/EX4200) that can do IPv6 routing in hardware (and ACLs), meaning that I can do things a lot more like I wanted with IPv4 because I don't need any NAT. One of the benefits for example is that I can just route a /64 to my local mirror servers over a VLAN for bulk traffic without putting any load on my firewalls. As the traffic is only to/from a few destinations and ports, ACLs in the switch are fine for me (+ local firewall on the mirror servers). This bulk traffic is probably close to 90% of my total bandwidth usage, which is just syncing the local mirrors. This means that this traffic is not clogging up the NIC queues on the firewall competing with "interactive traffic" like web-browsing, etc. Is this needed? No, not really. The firewalls can easily handle the traffic, but it is simple to remove 90% of the load on them for basically free.
All this may seem complicated and/or convoluted (and it probably is) but it makes (for me at least) my network much easier to reason about, and makes tinkering/experimenting easy and less likely to affect "production".
I should've moved to a dedicated home firewall sooner, but wasn't super high on my priorities. But with all the problems home modems/routers seem to be having, along with an upgrade to fiber internet that included a modem with hard coded default credentials that could not be changed I made the switch.
Only glitch I've noticed is a few problems with unbound dns, that I need to spend some more time on. DNS over TLS doesn't seem to be working for me, and I've had a couple glitches now when there are power fluctuations that I have to restart DNS after boot. But that could easily be something wonky in my setup and need to do more work to isolate the issue.
Overall, I would recommend for anyone who wants something a bit better than just the consumer grade stuff provided by an ISP, and for small business, remote sites, etc.
pfSense doubles the price over the exact same box you can get from AliExpress or ebay.
Buy a standard Ryzen mobo, a low-end recent Ryzen CPU, undervolt it, toss some ram in, an old SATA SSD, and a dual or quad port PCIe adapter off ebay. Or just buy an old Dell SFF PC; you can get your choice of how powerful a processor you want.
If you want the embedded/fanless setup, then buy one of the aliexpress/ebay boxes. Qotom is one such seller, I believe.
As someone with a ton of experience with opnsense: their release style is extremely irritating. The only way to get security updates is to conduct a full upgrade of the system. The UI is clunky, outdated, full of useless "help text", and almost seems to be purposefully designed to use confusing field names and terminology. Debugging problems is difficult at best with inconsistent logging infrastructure.
I've also never once had a config restore go properly, something I've tried to do several times because I generally get about 2-3 years out of an opnsense install before enough stuff has broken that I need to reinstall from scratch.
Last but not least: they now routinely ignore their support forum except for the most simple, common problems. Every time I've had a problem, I've found a corresponding post in their forum which has gone gnored by their employees.
Hardware remains one of the few viable business models for open-source software. The combination of features on this particular device have literally not been sold before, so in this particular case, the price may be somewhat justified IF they are also providing regular UEFI firmware security fixes.
With such high margins, it's surprising there are no hardware clones for AMD firewalls. Is that due to CPU shortages?
Protectli is a success story for Intel firewalls, where the open (coreboot) product earns a price premium and opaque clones expand the market.
If space or power efficiency don't matter to you, that's totally fine, then you can easily & cheaply match those specs. If you need something small and efficient, the markup isn't that large.
Nowadays, one can get a cheap Alder Lake Pentium + ITX/mATX with a cheap case, okay PSU, RAM, and SSD. That's no more than $350. Now new Intel dual SFP+ NICs are hard to find cheap, with Nvidia cards being easier to find cheap. So a dual port CX4 or CX5 from fs.com will run from $260 to $340. This still comes out ahead of the official HW. It all gets cheaper with older Intel HW.[1]
IMO, buying the official HW is officially supporting the OPNSense developers. As you noted, the DEC700 series is also quite small in terms of footprint. Those are definitely strong factors in favor of the DEC700.
[1] I mostly went with Intel HW, moving away from using a R5 3600 in that role. It seems that AMD's Zen2 is a bit less efficient vs contemporary and current Intel desktop platforms in idle to low-mid loads. Current setup with an older used Pentium and an used dual port X520 takes ~<20W on average. Though I doubt the DEC740 takes 15W at idle, probably less.
W.r.t. older Intel HW, I am thinking of 10th/11th gen parts, which can be had on good deals new.
They do? Show me.
But the value is crap. Essentially it's a SuperMicro 5019D-FTN4 with a worse performing chip, a built-in AMD SFP+ solution (which isn't as well supported as a cheap Chelsio card one can find on Ebay), and with no IPMI or normal VGA console. Which sucks. For something as critical as a gateway, it's really nice to just be able to plug it into normal rack console/screen/IPMI management for recovery and install. And originally barebones the 5019D sold for around $1k. While sadly amongst the supply shortages now they're more like $1400 despite being old, that's still ~$300 less than the DEC3850, and will end up about even with RAM and an M.2. But you still then get a faster CPU, and much better management. They throw in a single year of BE as a small sweetener but overall their offerings are straight downgrades in my opinion from just getting normal decent PC hardware. And that's part of the advantage of going to OPNsense in the first place.
So I kind of regret going for that vs just getting a normal SM system (various flavors of which I've deployed everywhere else, I got a bunch of 5018Ds for ~$600 for example). Not that setting up a special dedicated serial thing is a huge deal, but it's definitely an annoyance at that price level. Looking over the rest of their offerings it all looks similar: debatable quality for the money vs bog standard quality hardware.
And I want to be clear this isn't just a complaint about markup. I don't in principle actually mind paying more for the same thing if it comes with better support and someone standing behind it. The problem here is that the features are actively worse, and support is too! I was very surprised for example that something like Sunny Valley's Zenarmor actually has issues with the DEC3850's SFP+ that it wouldn't have with an old Chelsio card. So it's not like it runs OPNsense better.
Still, despite warts I'm very happy overall with the OPNsense, and with Deciso beyond their kit. It's also let me squeeze more life out of stuff I was feeling more iffy about (like UniFi and UISP for example, now I can just route their management VLANs via WireGuard for L3 management with zero internet exposure, and without UI's shitty ass routing/security I'm no longer feeling as pressured to leave ASAP).
I picked one up a few months ago as part of my attempt to overhaul my home network so I don't have to care again for a decade. Currently complemented by a "L3" 24-port Ubiquiti PoE+ switch (twelve 1GbE, twelve 2.5GbE, two SFP+) and an "L3" 32-port Ubiquiti SFP+/SFP28 switch. The hope is as DOCSIS 4.0 rolls out I'll get more value out of using SFP+ for a WAN port.
Grabbed a Micron MTA18ADF2G72AZ (16GB DDR4-2666 ECC VLP U-DIMM) to never have to care about memory. Ever. All the Suricata and CrowdSec rules in the world.
Also grabbed an Edimax N150 WiFi 4 USB adapter. This uses a RealTek RTL8188EUS chipset, which is extremely well supported by FreeBSD's urtwn(4) driver. So I use that to automatically do WAN failover to my iPhone 11 Pro as a hotspot if the cable itself went down.
Rock. Solid. And per my PDU, really just sits at 8-9W.
If my experience with this is anything like my experience with FreeBSD (limited) over the past 25 years, I expect the only downtime will be when I move house.
If anyone has any experience running 10G on these, I would love to hear more about it as my current ER4 is not going to be able to keep up.
The alternative would be to build a small x64 system with 10G but I'd like to find a solution that has low power consumption.
I had previously tried an ARM-based "Espressobin" which was only about $50 but had no 10G and little-to-no official support, couldn't even reliably boot into Linux.
Good signs for the future though!
hmm. There apparently is a successor "macchiatobin" that has 2xSFP+, but comments of poor software support is disconcerting
Other than that opnsense on an old ewaste HP SFF PC has been excellent. My only upgrade would be something lower power and fanless. Or to add it virtualised on my homelab R330, but security and no Internet when I break that wouldn't be a better situation than now
I suppose there isn't much profit to be made for a lower-end box, but I'd prefer to stick with *sense instead of wasting time learning Mikrotik's OS, even though they have some great price-points.
I bought my latest x86-based router from AliExpress, and never regretted it! Review available here: https://johannes.truschnigg.info/reviews/2021-01_fwbox/
If you want to use OpenBSD beware that you might not be able to push 1Gb/s with protectli devices. I had issues with OpenBSD pushing full gigabit, most likely due to all of the security mitigations.
I'm running a J4125 based topton box and it's running fine for about 2 weeks now.
But I’m happy with running some 23 VPS, a custom firewall that blocks QUIC/UDP and a transparent Squid that blocks DNS over HTTP.
All that for $80.
Sprung for some cheap spinning platters for a couple of separate RAID5s.
Oh, Proxmox is my choice of VM manager.
Only problem with netfilter/nftables is their inability to decremental IP TTL.
For example, I am using Unifi Wifi devices just for wifi, another box does all the NAT-stuff etc.
Somehow I fell into rabbit hole reading into how OpenWRT apparently is in process of changing how they do switches (related to bridging?), from homebrew swconfig to upstream DSA https://forum.openwrt.org/t/mini-tutorial-for-dsa-network-co... which seems to impact the setup a bit
A proper AP is usually an overkill for a home usage.
So is quad-core 2.2 GHz x86 router with 2x10Gbe, but here we are :)