Securing edge device systems, including firewalls, routers, and VPN gateways | Hacker News Reader

Securing edge device systems, including firewalls, routers, and VPN gateways

nsa.gov

78 pointstranspute36 comments

We need a SOHO replacement for APU2 routers: x86 open schematic hardware with coreboot open firmware, ECC memory resistant to Rowhammer, 6W TDP fanless, TPM 2.0 and DRTM secure launch. PC Engines was a Swiss company with Taiwan manufacturing, run by one talented human with a consistent ethos for ~30 years.

Since APU2 schematics are open, rebooting PC Engines in US/UK/AU could be initiated by industry leadership requesting AMD to restart production of the ancient AMD GX-412TC SoC, until AMD can ship a Ryzen Embedded alternative with comparable power efficiency. Ryzen Embedded includes dual 10GbE.

2023 EoL discussion, https://news.ycombinator.com/item?id=35635900 & https://pcengines.ch

  We hear all the time about promising kickstarters that fail and everyone just nods their heads in agreement that "shipping hardware is hard". And surely it is. But yet, here was one guy shipping thousands of custom-designed modern-ish computers to hobbyists and businesses around the world at a very fair price for multiple decades.

I think it wasn't just AMD winding down production of that SoC, but also that the Intel NICs being used on the APU2s (i210 and i211) were also getting hard to come by. Given how well designed and built those devices were, _especially_ at the price point Pascal sold them at, it's incredible to me that they're not everywhere. There really is no alternative, even at 2-4x the price point, and I'm surprised that AMD and Intel aren't trying to build more hardware to facilitate these sorts of devices, given how much pressure ARM has kind of put on lower power devices.

I definitely hoarded a bunch of APU2s when the final run was announced. There's just little or nothing you can get your hands on that works as well as they do.

Hopefully the re-industrialize [1] movement can inspire a new generation of board designers, learning from Pascal and research hardware like NetFPGA [2], which lead to commercial DPUs/SmartNICs.

[1] https://www.reindustrialize.com/resources/attendees

[2] https://netfpga.org

Why reinvent the wheels over and over again?

Just port coreboot, or something similar to the likes of https://cwwk.net/ which have done the hardware part sufficiently.

Or even better, organize to have some group/org paying them to port it to their HW.

Or/and get https://bootlin.com / https://www.collabora.com involved,

if they are disinterested, have them coordinate the effort.

They should know how to do that, instead of wasting time with reverse engineering all sorts of crap.

Apply same due dilligence to the firmware running on the NIC/switch.

Then continue with the 'optics' in the SFP+.

They are 'smart' nowadays, and often run Linux, or some RTOS, too.

See https://pon.wiki/xgs-pon/ont/bfw-solutions/was-110/#boot-log for just one example.

Diz iz dä äyge of phybre.

Äye vanna häff lain speed! Arr!

Do you know if CWWK x86 routers disable Intel BootGuard to allow coreboot? They have:

  capex: ~3X APU2
  opex: higher TDP
  manufacturing: China instead of Taiwan
  PCB design: China instead of EU

No, I don't.

But there is https://github.com/StarLabsLtd/coreboot/blob/24.12/Documenta...

which is basically the same shit with a different label.

So it should be possible.

CAPEX? Dunno what you are comparing these against? While I've never had an APU2, I considered them a looong time ago,

with 4GB RAM, and 4 NICs (or 3x & 1 SFP+) and just thought are you drunk/on crack/crazy/stupid because of the price at the times.

Which may have been because of greedy distributors/resellers in niche markets.

Completing them with cases, PSUs, and the rest of all the little things which go into them.

Those Changwan/CWWK/Topton/Whatever/lookalikes go for 200 to 300 Universal Credit Units with 32GB RAM, Case, PSU

depending on where, when, and how you get them. Aliexpress sales/coupons, or directly sourced from CWWK.

OPEX: Depends very much on the phase of the moon during design&assembly, the quality of their BIOS (Broken Initial Operations Setup),

the later mitigations of those by correctly configured most minimal Linux/xBSD, if SFP+ 'optics' are used or not(adds 2 to 2.5Watt per port),

borked kernelversions, 'smart' dynamic powermanagment not interfering with usage, multiplied by micromanagement of drivers, and so much more... …

(Speaking of N100 here, without coreboot or similar OFC)

Manufacturing: Why should I care?

PCB design: See above. Also economies of scale, mass production vs.'artisanal snowflake'.

Some more bait for you, since you're obviously interested, and I feel like it :-)

https://www.servethehome.com/tag/cwwk/

https://www.servethehome.com/tag/topton/

https://www.servethehome.com/intel-core-i3-n305-and-n100-2-p...

https://forums.servethehome.com/index.php?threads/cwwk-topto... (have fun ingesting 138 pages with comments and following up the links therein)

Also reddit #Alder Lake #N100 in r/homelab/minilab/MiniPCs/networking/HomeNetworking/HomeServer/SelfHosted/PFSENSE/whateverelse*

Edit: Also, because I felt like it:

https://yanling-store.en.made-in-china.com/product/zXPETvVub...

( https://www.aliexpress.com/store/1101257843 )

( https://www.ylipc.com/ )

vs.

https://eu.protectli.com/product/fw6d/

Scroll down -> Specifications -> Bios: BIOS AMI® or coreboot

Yawn... …

Where was I... …?

Never mind... …

( https://www.iwillminipc.com

https://www.alibaba.com/trade/search?spm=a2700.galleryofferl...

https://liliputing.com/this-small-fanless-pc-is-built-for-ne...

https://www.qotom.net/product/RouterPC_Q20331G9S10.html

Get the gist?

... … (to be continued by some other nerdsniped entity) )

https://www.servethehome.com/the-everything-fanless-home-ser...

https://www.servethehome.com/tag/qotom/

https://forums.servethehome.com/index.php?threads/qotom-denv...

Thanks for the Qotom Q20332G9-S10 pointer, a buffet of network and storage paths. Lots of innovative mini PCs out there, but it's hard to match all the APU2 checkboxes and price.

Intel finally relented a little on ECC segmentation to Xeon, with N97 [1] and i3-N305 CPUs having an in-band ECC controller. In theory, a NAS like [2] could use Ryzen Embedded V2000/V3000 with ECC, and a discrete TPM for measured OS boot with owner-defined keys.

As an existence proof of what's possible in quality compact hardware, SolidRun has a Ryzen Embedded fanless line [3] for industrial customers. As motherboards condense into a collection of SoC chiplets, edge hardware should continue shrinking.

On practical note with current hardware, a used Lenovo Tiny with Intel vPro and low-profile PCIe slot can have a quad NIC for routing, with Thunderbolt/USB4 to external storage for NAS usage. That includes TPM and DRTM, but still lacks ECC.

[1] ODROID H4, https://www.cnx-software.com/2024/05/26/odroid-h4-plus-revie...

[2] https://aoostar.com/products/aoostar-r7-2-bay-40t-nas-storag...

[3] SolidRun Ryzen, https://liliputing.com/solidrun-bedrock-r7000-is-small-fanle...

I'm ambivalent about the necessity of ECC in practice.

Especially since the internal pathways in CPUs already have it, and with DDR5 the DIMMs also have it, at least internally.

So that only leaves the path between RAM and CPU unprotected.

It would be nice to have, but maybe only to be able to feel good about it.

Depending on airflow, temperatures, quality of the contacts/slots/soldering, board layout, quality and

(more and more so with ever larger units on smaller process nodes) amount of RAM raising the probability of errors.

Regarding Lenovos tinies, I have several (dozen) M910q t(inies) with either core i5-7500t or core i7-7700t with 32GB (non-ECC)in use across my two homes, and am very pleased with them.

They run everything I've thrown at them without any errors. Giving the most clean kernel-bootlogs ever!

Cool&silent but still fast. Even most exotic stuff like Genode.

Really flawlessly working S3 (suspend to RAM) every time without exception, btw.

Can't tell for contemporary Windows, though, but don't care because no need.

Somthing like coreboot with the quality, correctness and functionality of their UEFI on cheap CWWK-like stuff would be a dream come true ;-)

> 2. Procure secure-by-design devices

I take this to mean "don't buy Fortinet products."

https://www.cvedetails.com/vulnerability-list/vendor_id-3080...

Cisco was in that list too.

Cisco ships so many hardcoded creds that you rarely need a vulnerability.

You probably should stop buying your favorite brand Palo Alto Network then.

https://www.cvedetails.com/vulnerability-list/vendor_id-1283...

Do you think we'd be any better off running SONiC?

Yes, but you'd run that on your switch and not your edge devices.

Palo isn't great either but Fortinet is in its own league.

https://www.theregister.com/2025/01/14/miscreants_mass_explo...

Yes, also Ivanti. And Palo Alto. And Cisco. And Dell (unless they spun that off already).

Most of the devices that rely on a scheme similar to inkjet printers (but with an even shorter shelf life) are going to be that way. This is because the money is not in the software, but in administrative choices (licensing, support contracts based on lifespan of hardware etc).

Since most deployment scenarios don't really need a proprietary ASIC to handle filtering, you'd almost universally be better off with a system that is built around generic white box hardware and an OS that is kept up-to-date. But that requires more knowledge and skills, and most people and companies would rather not invest in that for various reasons.

As for where you'd get your money's worth: it's mostly in the threat feeds. A well-tested, verified feed of known bad things (subnets, packet contents, behaviour) is much more useful than paying someone to keep a spare fan on the shelf so they can bring it to you "just in case".

The main thing the commercial players offer that open source doesn't do well is application level filtering. I want to be able to allow RTP across this giant port range but not just any UDP, or allow TLS exchanges with only certain SNI domains, not Cloudflare's entire address space.

If you want to do this, you need to select the least bad vendor.

In my experience, site categorisation is about the only 'feed' worth paying for.

Is application filtering always bundled with hardware? Open-source software already ingests URLs for adblock.

https://zvelo.com/about/company-history/

Opensource has dynamic RTP port opening based on SIP/SDP communication.

https://wiki.nftables.org/wiki-nftables/index.php/Conntrack_...

You can also send packets to userspace from nftables and do your SNI parsing/deep inspection/decision there. I used that a few times to do various things, like duplicate packet removal, etc.

It's very flexible.

The Lego pieces are indeed available for you go build this stuff yourself, but the engineering effort required to do so quickly make Palo Alto or Checkpoint's licensing look extremely cheap.

Yeah, until you hit some turd in fortinet (see how they mangle SDP if you send re-INVITE in a SIP dialog, even with all SIP protocol handling checkboxes disabled) and have to spend weeks with support and many hours of debugging and back and forth just trying to convince them they have an issue, after initially spending ~ 10h of dev/debugging time on trying to convince SIP phone manufacturer they have buggy SIP phone, before realizing different SIP packets are arriving on a SIP phone then are comming from PBX, because of this amazing forticrap middlebox. All the while whole company has issues with SIP telephony during attended transfers for months on end, disrupting commuincation with customers.

That shit pays for itself. :D

IIRC that's all under the NGFW umbrella, you can use things like zenarmor for that, it's essentially the 'paid feed' I was referring to, but as a plugin to existing FOSS firewalls.

Other useful feeds might be known malicious IPs and ASNs, dropping any packets matching those is very cheap and very effective. But they have to be reliable and not have false positives.

You could get a white box firewall put something like OpnSense business edition on it, and add Zenarmor. Works forever until FreeBSD no longer supports the hardware or until the hardware dies. And you get all the support and vetting/testing from those software options as well.

But realistically, if you're doing NGFW things you're probably in a compliance regime that doesn't allow for much choice of hardware and software and you're screwed anyway (compliance might require you to buy something like a Cisco or Palo Alto device + subscription, but then it turns out they run PHP as root under the hood and gets pwned monthly by a teenager on the other side of the world).

And TP-Link, Ubiquiti, Asus, Linksys, D-Link, Netgear, etc..

I think the only good options are something flashed with up-to-date OpenWRT, or a PC running something like Opnsense.

Yeah. Majority of their devices can be found on breached. I saw my ex employer on there. Not that hard to decrypt the hashes stored on the config files. Base64 from what I could tell.

You know you can build a firewall and/or router that has no IP address and cannot be accessed over the network…

A network slug[1], for instance, had almost zero attack surface.

[1] https://john.kozubik.com/pub/NetworkSlug/tip.html

Yeah but then you need an oscilloscope to really know whether the NIC firmware is phoning home.

Joking of course (but also not).

What’s the difference between this and a managed switch with firewalling capability?

Anything that you can buy off the shelf is compromised.

I use OpenBSD on all my edge devices. It's not perfect but it is superior to 99% of everything else. That combined with poisoning the replies to nmap scans (fingerprinting) puts me in the 'much harder' to compromise category.

"Security through obscurity" isn't security. But "Don't be where your enemies expect you to be" is still good advice.

Also, relying on 1 layer of security is insanity. You need multiple layers, you need isolation.

I highly recommend OpenBSD for firewalls, vpn (wireguard), and other edge servers. It has served me well. I love how everything is organized.

The only caveat is limited WiFi support, essentially not 802.11ac or later, so you will need a separate AP.

Most wifi driver firmware is full of remote exploits, and still should never be connected to secure LANs. =3

Yes. I used a Huawei 5G router as cellular backup on my home network OpenBSD router/firewall but I ran Wireguard over it because I trust neither Huawei nor my cellular provider.

I'm not quite as hardcore in that I use OPNSense (FreeBSD-based), but I still rate that as a good level above consumer-level and ISP-provided modems / routers.

I'm not sure what vulnerabilities I have from the Ali Express multi-LAN-port hardware that OPNSense runs on, but I don't have the motivation / money for nation-state level paranoia.

The myth BSD is more secure comes from the frequency of unreported zero day exploits. One of my former managers ran the platform for years... right up until the institutional router was compromised.

The lesson here is that if the device was made by an external firm, than the responsibility wouldn't have fallen on him politically. One may assume it is operator/administrative error, but this guy wasn't YOLO'ing by any stretch of the imagination. He was replaced 3 months later for unspecified reasons.

Critical systems are a different problem domain with different rules. =3

> Securing edge device systems, including firewalls, routers, and VPN gateways

Which they made them insecure by mandating all kind of backdoors. See Cisco, PaloAlto, AMD PSP, Intel Me, Apple hardware backdoor.

Reply on news.ycombinator.com