OpenBSD on EdgeRouter Lite | Hacker News Reader

OpenBSD on EdgeRouter Lite

tedunangst.com

112 pointsingve27 comments

That router uses as CPU a Cavium Octeon processor (MIPS architecture) with two cores. It supports MIPS 64 r2 and aligned data access, and it is a great platform for running big-endian unit tests.

It boots from a internal USB drive, that can be removed and changed (the only problem is that not all drives are compatible, and also, in most cases you will have to the pendrive in order to use it, not a problem).

I bought it not long ago for doing big endian on both 32 and 64 bit modes, and I can not be more happy. Amazing stuff. My intention was to put directly OpenBSD, but the default Linux distribution it uses (Vyatta, Debian derivative) works very well, being able to install packages via apt-get, so I use it too. To be able to run OpenBSD just changing the USB stick is amazing, and great for testing without spending lots of money on more expensive stuff (e.g. while there are many affordable, and quite good, routers with fast single-core MIPS 24K and 74K -e.g. TP Link-, usually have small flash disk (4-16MB) and low RAM (16-64MB), having to use remote filesystems or cross compiling; with the EdgeRouter Lite you can build and run, without worrying with cross compilers nor remote filesystems).

Amazing hardware: USB boot means no device brick risk, multiple OS boot. Also, is very stable, no matter if it gets a bit hot, I had 0 hangs since I bought many months ago. Despite being not cheap, compared to other domestic equipment, it is worth it every cent it costs. In my opinion.

P.S. As contrast, another device I use for unit testing, a GuruPlug (ARMv5), has the risk of being bricked, make me being afraid of installing anything not really necessary (the point of using an ARMv5 instead of an ARMv6 like Raspberry Pi, is that the Pi supports unaligned memory access, so if you want to check those cases, you can't).

Do you use the router features as well, or do use it only as a mini computer for testing?

Sure! As router as well, simultaneously. I access via ssh, having repositories inside, doing pulls, builds, etc. Is just like any other machine (in that router you can run at least Linux, FreeBSD, and OpenBSD). Network performance is great. I use two ports for two internal networks (through a gigabit switch in one port, and the second usually used only for specific experiments), and the third one for the WAN access.

Hardware is fast: two 500MHz MIPS64 CPUs with 512MB of RAM (Cavium CN5020, 128KB L2 cache, 64-bit memory transfers are faster than twice as fast as 32-bit ones), each of one doing 2 instructions per cycle (4 instruction per cycle, counting both CPUs). The only notable downside is not having hardware floating point support (software doing extensive floating point usage will run slow).

The device has hardware packet and cryptographic acceleration. In the Linux case is able to fill gigabit links (manufacturer claims 1 million packet per second -I haven't pushed it that far-), and also some features are available on FreeBSD (e.g. hardware crypto, but not packet acceleration, if I recall correctly), and few/none (?) on OpenBSD (uses only one half of the RAM and just one CPU).

Thanks for the extended information! :-)

It's probably much too late for this to be helpful, but the issue with not all flash drives being compatible is because that version of uboot would probe for USB devices very early on and it doesn't probe again if it doesn't find anything. Changing bootdelay won't fix that because that happens afterwards. It can boot from practically any flash drive if you prepend "usb reset;" to bootcmd and then do a saveenv. Be careful when doing this, all of that configuration isn't stored on the flash drive obviously so if you aren't comfortable with uboot you can easily brick the router. Just do a printenv first and make sure to copy down the old bootcmd just in case you need it to recover.

Thank you for the explanation. That is not really a problem, as there is information about drives known to work (you can reach it e.g. via Google). Also, USB pendrives are cheap, so just trying with pendrives you already have it is probable you find many already working properly.

wolfhumble has a dead question [0]:

"Do you use the router features as well, or do use it only as a mini computer for testing?"

Given that the ERL requires a blob to use the hardware offload features of the Octeon (without which routing and firewalling performance is reduced by a factor of 10 to 20), one would expect that using the thing as a router while running OpenBSD would give you -at best- 100mbit/s throughput.

Does that match your experience, or are you not really using it as a gigabit router, or am I woefully misinformed?

[0] According to his comment history, it looks like he's being shadowbanned, but I can't figure out why.

Thanks for the heads up about the shadowbanning, now removed. It was caused by a moderator misclick or other mistake. Thanks again! :-)

No problem!

I found myself mistakenly shadowbanned for a while quite a while back. I wonder what the UI for performing that action looks like.

How does this compare to a PC Engines APU w/pfSense?

http://store.netgate.com/kit-APU1C4.aspx

http://bsdrp.net/documentation/examples/forwarding_performan...

I guess the EdgeRouter is cheaper at least.

The PC Engines APU is great. Soekris is another hardware company making high-performance BSD-compatible routers:

http://soekris.com/products/net6501-1.html

Does not support the hardware offloading though, so you're not getting the advertised processing speed.

But, as with RPi, you get to boot your own choice of OS.

In my opinion the advantage of this (=more control) easily outweighs tradeoff of not "opting in" to hardware features enabled via binary blobs.

A MIPS board that pretty much "just works" with BSD kernels. No brainer. I'll take it.

If this is your goal, then EdgeRouter X would be a much better choice. It has a faster CPU but no offloading, so it would outperform the Lite in this use case. The MSRP is also half that of the Lite.

it wouldn't because at this time there is seemingly no alternate operating system that supports it.

$91 for a router with a few mb of ram?

with some $20 more you can build a amd 5350 or a Intel j9200 with some 4 port pcie network card and have any os on a fully capable computer, probably using less power than that thing.

The hardware offloading (which seems to be missing with OpenBSD) is what makes that a great piece of hardware for the price.

Can you point to the parts on how to build the system you describe for $130?

Less power? The j9200 has a TDP of 10W and an average usage of 8W [1]. The entire ERLite unit, not just the CPU, has a maximum power draw of 7W [2]

As others have mentioned however, network offloading on these units is the selling point for these machines. It's also a great value for those looking explicitly for a MIPS or big-endian arch.

[1] http://cpuboss.com/cpus/Intel-Pentium-J2900-vs-Intel-Core-i3... [2] https://dl.ubnt.com/datasheets/edgemax/EdgeRouter_DS.pdf

Though they have little in common besides "O-P-E-N" in their name, it's worth noting that OpenWRT works on a lot of Ubiquiti products, too.

Doesn't Ubiquiti use openwrt as their base?

I believe vyatta is their base.

Not always.

But for the product discussed here it is. (They actually hired some of the Vyatta devs to work on it, since the upstream project seems to be left to die at Brocade)

since the upstream project seems to be left to die at Brocade

Indeed, Brocade closed sourced the main product in 2012 and left the community edition to die.

Thankfully the Vyatta core was forked in 2013 (renamed VyOS) and is under active development as a free open source project. [1]

[1] http://vyos.net/wiki/Main_Page

I think it's v6.3 of vyatta they've forked. Vyatta being debian based, so is EdgeOS.

There's also this conversation regarding Ubiquiti's GPL issues from the past:

https://news.ycombinator.com/item?id=9331512

I've for three ERL's (one at home, two elsewhere) and they've been great so far. But I run their firmware, not OpenBSD, so I can't comment about it much. I also use their APs and I'm thinking of getting their IP cameras.

Reply on news.ycombinator.com