The ultimate OpenBSD router | Hacker News Reader

The ultimate OpenBSD router

bsdnow.tv

166 pointsfcambus79 comments

That's a very ambitious take on this problem. It is pretty cool (and definitely more secure) to run OpenBSD but you can probably get most of the upside by slapping OpenWRT on the consumer router you already have. A $50 WDR3600 happily handles several VPNs, custom VLANs, an IPv6 tunnel, exotic routing, an external drive, and a Samba server while doing the typical SOHO router-y stuff, like wifi.

BTW Running my own name server has solved a lot of weird slowdowns I used to experience when browsing the web or sshing. According to namebench[1], my router doesn't even crack the top three when it comes to response time so I used to have it forward queries but in practice, after it warms up, it's more reliable and delivers a smoother experience than either my ISP or Google.

[1] https://code.google.com/p/namebench/

We're already seeing some large router makers (slowly) adopt OpenWRT/DD-WRT. Maybe there's a chance we can push for a status quo on using open source firmware for routers - perhaps some sort of "Android" for routers, but it needs to be fully GPL so OEMs can't lock down 95% of it like they do with Android.

Specifically, GPLv3 or AGPLv3 to avoid consumer device lock down.

A typical $50 consumer router running OpenWRT will do everything you want except traffic shaping. The cheap consumer stuff all uses slow single-core MIPS CPUs that can only do traffic shaping at tens of megabits and consequently cannot do QoS for a fast DOCSIS or fiber connection. Of course, even if you do go to the trouble of getting an Intel-powered router, you still wouldn't use OpenBSD (or anything other than Linux) for QoS.

  > you still wouldn't use OpenBSD (or anything other 
  > than Linux) for QoS.

Is this still the case? Especially in regards to OpenBSD managing a fast residential cable connection? I thought 5.6 yanked out the slow ALTQ code and brought OBSD close to linux QoS performance, especially for residential traffic management.

From a quick skim of the manpages and a few google searches, it looks like pf is a really poor substitute for tc. Without CoDel and friends, OpenBSD can at best implement half of a good QoS system.

Your categorical rejection of openbsd+pf is based on a cursory google search and man page comparison? I have not seen any QoS comparisons of OpenBSD 5.6 versus Linux. Did you find any?

There's nothing to compare. OpenBSD just plain doesn't have any active queue management. They used to have RED in the altq module, but it's been removed and I haven't found any mention of any other form of support for any of the common AQM algorithms. If it supports anything more advanced than classifying packets into a fixed set of priority queues with fixed packet count limits, it's well-hidden. What functionality is exposed and advertised through their man pages is simply not enough to put together a fully functional QoS system, regardless of how efficient it is with CPU time. There isn't even the theoretical possibility of OpenBSD doing good QoS unless they've got a large amount of complexity hidden and misleadingly glossed over by their documentation. In this case, running a dumb algorithm arbitrarily fast can never compete with the smarter algorithms.

Might I suggest that you try it out before making such a firm statement?

Try what? The pf.conf man page section for queuing describes only functionality that is effectively equivalent to HTB or possibly HFSC on Linux. That algorithm on its own fundamentally cannot keep latency low because it has no mechanism to restrict queue length without crippling bandwidth. If OpenBSD's current HTB-equivalent is much more efficient than it used to be, that's an improvement but it's still only one of several required components and cannot get the job done on its own.

I second that motion. Try it (OpenBSD with pf). I have not played with traffic shaping for myself but as always with OpenBSD I guess i would be surprised how well it works. set prio, set queue and set tos seems to be the "tools" to use.

Why should I expect OpenBSD to not suffer from the universal problems of overly large queues when they haven't documented any mitigation techniques? I can't even find out how to get OpenBSD to do ECN marking without the now-removed altq module.

Does this help https://calomel.org/pf_config.html?

Why not?

i'm too lazy to search, but almost every cheap modem OS and even windows and OSX were susceptible to pings or some other connection completely blocking out all honest traffic.

Maybe it's gotten better in recent years, but last time I tried consumer hardware (Linksys), there were actual hardware problems like overheating that didn't get any better with alternative firmware. My Linksys routers (I went through three of them before giving up) would lock up with increasing frequency as they aged, under both Tomato and DDWRT. When I went to Cisco (real Cisco, not Linksys Cisco) and Ubiquiti routers, the problems went away. For the price of a Ubiquiti router (under $100), I personally find no reason to spend time with open source firmware on crappy consumer hardware. It just works and is rock solid.

It can be both, but certainly one issue is hardware quality. The Linksys WRT54G and WRT54GL (of which I've purchased 8), are cheap electronic devices that last on average 12-18 months. I did have one make it out to 3 years, but I also had one kick the can about 6 months in - so it averages out. What's worse, is the failure case is "Intermittent performance" prior to just dying.

Cisco also has made crappy hardware in the past, I've had stacks of Cisco 2960 switches with dead ports, but they make up for it on the business side by having really good RMA processes if you have support agreements. They also make phenomenal hardware as well - I had a Cisco 7206 keep running without any human involvement whatsoever for 10+ years.

I'd be willing to pay $500-$550 for a solid 4 Port GE, 1U Rack Mountable, fully open, appliance that I could drop whatever operating system (OpenBSD preferred), that had gone through some extensive HALT (Highly Accelerated Life Testing) at temperatures -55C to +85C that suggested an average of 20 year life in typical consumer (23C) environments. I bet there are 10s of thousands of people who likewise would jump at the opportunity.

Sometime I wonder why the WRT54GL still get's hyped so much. Yes, it is probably the router that started the whole Wrt projects. But seeing that this 2002 relict still sells for 50+€ (on amazon) pains me. For the same money you get a superior device (eg a tp-link tl-wr1043nd and even that one is quite old by now) that runs Wrt just fine.

Probably for the same reason we buy these: http://www.amazon.com/USRobotics-USR5686G-Serial-Controller-...

They basically get a (minimal) job done. (Though, unlike the WRT54GL, the USR Robotics modems are pretty reliable).

What I don't understand, is why nobody ever built a somewhat robust version of the WRT54GL?

Just build a robust device with a Skyworks 802.11a,b,g,n,ac FEM and maybe four GigE ports. Something plain, vanilla, that you can stick on your desk for 10 years. I bet they'd sell millions of them.

> a superior device (eg a tp-link

My experience with TP-Link has not been that great. I have three TP-Link devices (wireless router, Gig-E switch, and wireless AP) and I've had stability issues with all three since I got them. I regularly have to reboot each device, as each one will inevitably lock up after being powered on for a few days. I've also tried open-source firmware on the router, and it ran terribly no matter which flavor I tried; my 25Mbps WAN connection was reduced to about 5Mbps across the board. It runs at the correct speed with the stock firmware.

I'm seriously considering a build like the one in the article, however I'll probably go with a dual NIC Atom based nano-ITX board instead. Not only will it be cheaper, it will be easy to repurpose as a full fledged PC if I upgrade routers again in the future. A good example is this complete machine for $250, case and all: http://www.amazon.com/dp/B008KB5YCK

With regards to the name-server bit, I honestly can't re-iterate this enough. It's amazing how many "My internet is slow" remarks really come down to DNS resolving slowly. My router has WAN failover, and I've had to manually tweak the DNS it chooses many times because even when a WAN is 'up', the DNS it uses might have latency exceeding 2+seconds for hours at a time (Comcast) even though the link itself is perfectly fast (125Mbit).

Merely setting a different DNS source immediately shows off the full potential of the link. I still need to setup my own local DNS server so that I can be rid of this problem entirely. :(

So... $337 for a decent soekris board and 'ok' case... or $340 for this: http://www.amazon.com/Ubiquiti-Networks-Edgerouter-Router-ER...

The latter has a complete open-source OS, you can ssh in and re-flash it yourself easily, a great community, the same TCP hardware offload, etc. I have been spec'ing out a BSD+soekris board setup for years, but when the Edgerouter came on the market it was a no-brainer. The fact that it works-out-of-box with little effort (for someone experienced with networking) is a big win, and that its quite easy to re-flash it and tweak as desired sure doesn't hurt.

While I really dig the DIY-router stuff, and was about to do it myself, Ubiquiti has sure made it hard to go that route when they can supply dang good products for the same price or less.

Edit: Added bit that this isn't a "zero effort for newbs" type product. If you've never setup a router, there'll be some research in your future to setup an Edgerouter, or BSD router.

I just use the same hardware (Soekris net6501) with pfSense 2.2. Still requires a little more work than the Ubiquiti, which I might try next time. But it works pretty well 'out of the box', too.

Different strokes for different folks. The Soekris hardware has a reputation for being industrial-strength and very resistant to harsh environmental conditions. The board requires no fans (the Ubiquiti has them) and has Intel NICs. The Soekris case fits nicely in a small cabinet in the average branch office and you don't need a proper cabinet for it. Disadvantage for some but for my little office, the Soekris is perfect. The speed of the Soekris is completely sufficient for our 100 Mbps biz class cable.

It turns out a lot of the processing is offloaded to proprietary drivers on the EdgeRouter board. It's great for running their stock firmware, but not so great running OpenBSD. Ended up going with the new PC Engines apu1d4 board (http://www.pcengines.ch/apu.htm) rather than a Soekris. I believe it's the best of the options for an OpenBSD router.

I've been using the APU board with great success at home for a good while. I upgraded from the Alix board to make full use of my 1G/1G broadband.

Doing some speed tests I've seen speeds fluctuate around 500/700. Never really reached 1Gbit on any public speed test yet but it's helluva lot better than my old alix router.

I love the fact that the APU has an open bootloader, and as far as I can remember it was cheaper than what is mentioned of soekris here. I seem to remember the whole package costing me around 100 eur.

I'm running pfSense on the apu1d4 at home and all is well.

This is a seriously awesome podcast. Consider listening to it, the amount of knowledge combined with two bsd-loving hosts is amazing!

I do love how they go back and update their tutorials. Given how much has changed since the original version, its really nice.

In my experience, everything from Jupiter Broadcasting [0] is extrememly top-notch and informative. I'd highly recommend all of their shows (including BSD Now!)

[0] http://jupiterbroadcasting.com

Yeah, I've been Jupiter listerner for some time now mainly "Linux Action Show" http://www.jupiterbroadcasting.com/show/linuxactionshow/ they have a lot of useful info. Highly recommended podcast, and if you can support them with anything you can. I'm also thinking on building my own next DIY router. I'm tired of how limited the routers you buy off the shelf are. Currently running Asus RT-N56U with Padavan F/W https://code.google.com/p/rt-n56u/ I've also always wanted to switch to BSD distro but ports are not updated as often as "Arch Linux". If there where BSD rolling distro similar to well updated Arch Linux packages, I would consider switching.

You can use pkgsrc current with any BSD if you like.

Which ports system? I can't say anything about OpenBSD ports, but aside from some stuff that gets very little love from people, most of the FreeBSD ports tree is kept bang up to date, and binary packages appear shortly thereafter. pkgsrc, OTOH, is only released quarterly, though if you want, you can sync with their CVS repo, though, y'know, CVS.

The BSDs aren't distros, though some do have what might be called distros, such as PC BSD, pfSense, &c. being distros of FreeBSD, EdgeBSD being a distro of NetBSD.

The ports system is a rolling release system for non-base software though, though the base OS isn't. The closest BSD to come to having a rolling release schedule for the base OS is OpenBSD, with its six-month release cycle. The thing is that the BSDs can't have a rolling release schedule as is found in some Linux distros because the base OS is managed separately from the ports/packages: the core OS components aren't packaged, so there's no sense in which they can 'roll'.

Personally, I'd never use an OS with a rolling release cycle on a server. Too much can go wrong.

BSDNow has some great sponsors too.

I see what you did there. Tarsnap is awesome indeed :)

Personally I like to get a consumer router and put OpenWRT on it. It used to be a lot harder but it's gotten a lot more simple and effective. I have a few reflashed Netgear WNDR3700s but there are probably better ones out there that are pretty cheap too.

It takes more research and work but it's more simple than having to install everything onto a clean OS install.

Does Soekris have any competition in this space? Any time I check I can never find any viable competitors. Soekris seems to have hit the "IDA Pro sweet spot," AKA unbelievable product priced just low enough to scare away any new competitors.

Actually, soekris were pretty good but they are kinda outdated. I just got an apu1d4 from pcengines[0] and OpenBSD works like a charm on it. Half the price, much more power.

[0] http://www.pcengines.ch/apu1d4.htm

You need to look a bit closer to see the difference. APU uses cheaper realtek NIC and soekris uses Intel's. Soekris gives Intel QuickAssist to help encryption (VPN etc.). And soekris is showing up a new model later this year: http://soekris.com/products/net6801.html

But PC-engines boards are nice for their price. I'm considering to get one at home. That or spend a bit more on soekris box/

damn. you people are like audiophiles for network. ...is there a name for that already?

those two boards have more CPU power than my home theater PC... which crunches 720p video all day long without a decent GPU.

This is nothing like the absurdities of the audiophile community. Routing traffic just isn't as trivial as you think it is, and going with cheap NICs that are missing important features and have lower-quality drivers will produce measurable and significant differences in objective benchmarks. If you have something like a 100Mbps cable connection, you can't use just any off-the-shelf hardware and expect to solve any problems in software.

Thats what an audiophile would say

;)

I'm going to re-quote the parent for you.

> Routing traffic just isn't as trivial as you think it is

---Alex

It's also what the IETF says. Don't belittle those who are trying to fix the problems you're frustrated with but can't be bothered to understand.

The NIC used is also important if you need the router to do PPPoE. Most Intel ones will support jumbo frames which can let you still get 1500 byte packets to the ISP.

PC Engines, Netgate.

I think the Netgate C2758 looks suspiciously like a rebranded SuperMicro 5018A-FTN4 (of which I own two)...

You're right, I don't see the point of Netgate C2758.

However, this is a different design: http://store.netgate.com/ADI/RCC-VE-4860.aspx

The lower-end PFSense rebranded hardware [1] seems pretty competitive to me, with the $300 box they're expecting to release at the end of June looking especially nice in comparison to the soekris boards. Alternatively, if you're willing to spend a few dollars on RAM, there are a number of barebones systems out there with Atom C2[3,5,7]58 processors that will give ten times the performance, so you could do some file serving with the box too.

[1] http://store.pfsense.org/hardware/

HP MicroServer with a second ethernet card. There are usually cashback deals on them and mine came with a HD. They have ECC ram too.

Dell T20 Xeon with a Type-1 hypervisor to run the router VM, using VT-d to isolate a multi-port NIC to the router VM. The same box can run other VMs with FreeNAS, Linux, Windows, etc.

I guess I should have been explicit and stated I was looking for a comparable system that competes with the soekris. Does the dell t20 really count as competition to a soekris? A quad port intel NIC and a Dell T20 w/ Xeon is at least twice as much as this soekris. And then there is the whole issue of size:

             Dell  Soekris
  Width      6.89    11.40
  Height    14.17     1.41
  Length    17.15     6.11
  Volume  1675.31    98.21

It depends on whether one also needs a NAS with RAID disks, or a Windows desktop, or a media server, or a personal cloud. If you need any of those, then there is nearly zero marginal size or cost to add a router VM, with more hardware headroom and OS flexibility than a special-purpose router.

Some supermicro mini-ITX boards (some work straight from 12 V)

some other mini-ITX boards by random manufacturers in that space

Lanner has some interesting Atom-based options. I'm testing the FW-7525A right now, which has physical hardware bypass on one pair of ports.

Where can you buy Lanner products? I emailed them in the past regarding 7525, and they refused to sell me just one unit.

Mine is through Pyramid (www.pyramid.de) who resell some Lanner stuff. Don't know if they'll sell outside Europe, and can't guarantee they're interested in selling single units to new customers. (We've had some previous business with them).

And for the rest of us there is: http://routerboard.com

Which as-far-as-I-know comes with all open source software, it very well supported by a large community.

That's the Mikrotik hardware, and I don't believe RouterOS is open source. The "L3", "L4", "L5" you see on the product listings are the RouterOS license levels which control available features.

You can also run OpenWRT though installing it is described as "Not straightforward."

I've been using MikroTik lately because I wanted to identify my traffic and Qos it differently for VoIP installations. It seems to do well with this.

Allow me to save all of you who follow this guide $14 and hours of headaches: don't waste your time with the internal USB port on the Soekris net6501. The little Sandisk Cruzer drives that fit inside the case are total crap. The two that I bought lasted less than a day each. I think that writing the 4GB PFsense image to them was enough to kill them. Unfortunately for me, it didn't kill them in an obvious way. In my case, strange things started happening in PFsense. DNS became half-broken, DHCP for new clients didn't function, etc. I finally realized what was wrong and threw the USB drives in the trash and bought some of these guys and the problem was solved:

http://amzn.com/B00ELQZD10

Most consumer routers include an access point too nowadays.

The downside of this OpenBSD setup, is that you still need a consumer grade AP next to your router (that's exactly the setup I have).

OpenBSD still doesn't support 802.11 > g, regrettably.

I'm using an ASUS RT-AC68U with totally open source firmware, supported by asus, with timely updateds to fix security issues. It's a 802.11ac. I really recommend it. Works very well.

Merlin FTW

https://wireless.wiki.kernel.org/en/users/drivers/b43 says that the BCM4360 802.11ac chip in that is unsupported. So what open-source driver are you using for that?

I just added an OpenBSD firewall in bridge mode between my Comcast router and the rest of my network. It's implemented on a Shuttle DS57U (dual-core 1.5GHz Broadwell Celeron 3205U), with 16GB Crucial DDR3L RAM and a 128GB Crucial SSD (leftover parts). Total price: $358. It's a pretty sweet box: fanless, metal chassis, dual Intel Gigabit Ethernet. The only (minor) quibble is the CPU doesn't have AES-NI.

Thats an aweful lot of work to roll out your own router. What does this have over, say installing pfsense to a box?

Save money on power cost for one! If you installing pfSense consider small booksize or itx case.

Neither one will be much different if the hardware is the same.

First thing that comes to mind is having more control of which version of *BSD you want to be based on. With pfsense you're on whichever version of FreeBSD they've worked up to.

That said I roll pfsense on a thin mini-itx intel board and its great.

Here is the link to the relevant section ...

https://youtu.be/a-wtYUKoBa0?t=3662

Is DNS caching still relevant these days (since we have fast connections now)?

Bandwidth has gotten better, but latency hasn't.

Yes, unfortunately the speed of light has remained constant over recent years.

>Atom E6xx series processor

My only concern would be the use of a SSD for something like this. I know with PFSense and Untangle, applications like this will shred a ssd in fairly short period of time.

Are you writing more than 200 TB of data? http://techreport.com/review/25559/the-ssd-endurance-experim...

The only thing such a system should write in bigger amounts are logfiles, unless you keep very detailed logs/packet captures you shouldn't get close to the write limitations of a modern SSD, which is tens of GB per day. (You might want to change the fsync-behaviour though to limit write amplification, e.g. syslog syncs the logfile to disk after every log line, which might become a problem if you have very active logs?)

I'm not sure of the exact details. I've only done a couple untangle boxes (using NexGen appliances). 6-8 months ago colleagues that do a lot more of them were still seeing a much higher failure rate across all SSDs than with spinning disks in Untangle boxes. Although I see now that NexGen offers a SSD Option on their boxes now. so things might have improved.

Reply on news.ycombinator.com