The Ars guide to building a Linux router from scratch | Hacker News Reader

The Ars guide to building a Linux router from scratch

arstechnica.com

166 pointsthehoff88 comments

If you don't want to do all that configuration, PFSense is a good alternative. https://www.pfsense.org/download/

PFSense is the same thing below the hood, but with a web front-end and plugins.

Most off the shelf wireless routers work fine as an access point, but are quite bad as a router. So you can just plug your old wireless router into this thing (with DHCP etc turned off), and your whole setup will be much better.

I'll echo the pfSense recommendation. One correction, though: pfSense is built upon FreeBSD, not linux.

IPFire is my choice if you prefer Linux.

I've been running pfsense for over 5 years now and never had an issue. And, it's only getting better. I highly recommend it.

I built a lan to wan router where the wan was a wifi link (couldn't run a cable in the building) using pfsense, it works really well.

Nth-ing pfSense. Straight-forward to configure, dead stable. Near-enterprise class features.

Ubiquiti's EdgeRouter Lite is a popular, fast, cheap (<$100), solution in this space.

People are running FreeBSD and Linux on it:

http://www.daemonology.net/blog/2016-01-10-FreeBSD-EdgeRoute...

https://wiki.gentoo.org/wiki/MIPS/ERLite-3

One note is that it's my understanding that running Linux or even FreeBSD will mean that you can't hit 1Mpps if that's important to what you're doing.

The driver for the network offload is proprietary, however the pre-installed OS is Debian Linux with a fork of Vyatta.

Excellent point. I should have been more specific.

So you go through all that trouble and end up with a terribly slow NIC, a "grownup" OS to maintain and the shitbox MIPS architecture to boot? What a pain. These MIPS boxes have their place, and that's squarely in the OpenWRT / system builder niche.

With the electricity prices the way they are in the US, no way I'd run something like that over a proper x86 as in the article that can saturate a 1Gbps and manage decent disk IO.

>>a proper x86 as in the article that can saturate a 1Gbps

"x86" covers a lot of ground. Certainly, in the "about $100" range, your choices in X86 land can't saturate 1Gbps either. The x86 box in the article was $280 total. About the same cost as the logical upgrade to the EdgeRouter Lite...their ER-8, which can saturate 1Gbps, and has 8 ports.

Can 2nd that its a very powerful and cheap router that could solve all my needs out of the box. Like routing my tv through a vpn etc.

Here's a tutorial for doing the same thing with OpenBSD from a few years ago: http://www.bsdnow.tv/tutorials/openbsd-router

The OpenBSD site has an example as well.

http://www.openbsd.org/faq/pf/example1.html

As a point of comparison, the iptables syntax as shown in the Ars article is far harder to grok at first glance than either of the pf examples. For example:

    pass in on egress inet proto tcp from any to (egress) port { 80 443 } rdr-to <ipaddr>

vs:

    -A PREROUTING -p tcp -m tcp -i p4p1 --dport 80 -j DNAT --to-destination <ipaddr>:80
    -A FORWARD -p tcp -d <ipaddr> --dport 80 -j ACCEPT

Yeah, I've always found iptables to be a little hard to understand.

It also sounds like it's getting replaced[0]. My guess is that we will see iptables around for a long time after it's been deprecated. ifconfig, for example, is deprecated[1] yet it's still around and being used.

[0] https://lwn.net/Articles/564095/ [1] https://lists.debian.org/debian-devel/2009/03/msg00780.html

ifconfig is BSD derived and standard across more Unix variants, whereas ip and friends are Linux only.

See also dladm in illumos as another variation on a theme.

ifconfig is still used on BSD and, most importantly, MacOS X. Also, the Linux replacement for ifconfig is ip from iproute2, and that was made by the guy who did Linux QoS. He sort of sucks at usability.

For example, `ifconfig eth0 down` becomes `ip link set dev eth0 down` and `ifconfig eth0 192.168.1.2` becomes `ip add add 192.168.1.2/24 brd + dev eth0`; I had to look these up to make sure I had the correct syntax.

Okay, the actual command is `ip address add` and not `ip add add`, but it allows extreme abbreviation at the expense of discoverability.

i would say that openbsd is the better tool for the job in my opinion. pf is just much nicer and simpler to set up and configure than linux iptables, and openbsd is hardened which is important for a machine that is always facing the internet.

Another thing that's nice about the BSDs is that they are whole systems, not different sets of kernel+utilities packed up by third parties.

Generally speaking, that means that there's one canonical way to do something instead of a bunch of different ways like in Linux (think 'ifconfig' vs 'ip'). Of course, FreeBSD ships with three different firewalls[0], so that's not always true.

There are downsides, of course. A freshly-installed BSD has a lot less stuff than a freshly-installed OpenSUSE/Ubuntu/Fedora/etc.

[0] https://www.freebsd.org/doc/handbook/firewalls.html

i don't think that the minimalism of the default install (i am talking about openbsd mainly, since i have little experience with the other three) is too much of a problem. in the default install is practically all the tools and components one needs to set up and configure a router. any extra components can be grabbed from ports and packages as shown in the aforementioned tutorial, where the dnscrypt client is installed for encrypted dns queries.

I'm sad they don't use dnsmasq in the article, it's a ton more easier to setup than bind and even commercial routers use it.

Yeah, plus it can also serve as DHCP server, one less package to install/maintain.

I wonder how well the Turris Omnia[1] will compare to a homebrew solution like this.

[1]https://omnia.turris.cz/en/

I cant wait for mine to arrive :)

I like those automatic security updates and network traffic analysis...

I had trouble getting a hold of one of those C1037U boxes from China. The seller would 'run out of stock' frequently if I found one for a decent price.

I ended up going with the APU2B4 board (an upgrade from the APU1D mentioned in the article.) I put pfSense on it, and it's been running perfect for a few weeks now.

Even that board is probably massive overkill for most people. I have 50/50 internet, and with full bandwidth used by torrents, a VPN and ssh session open to the router, and the web interface open, I'm still only getting about 10-15% CPU.

http://pcengines.ch/apu2b4.htm

Looks like the one listed in the article is both out of stock and going for ~$1400 (I'm guessing auto pricing?).

The APU boards are nice too; I thought about going for one when I was shopping for a better router and slapping OpenBSD on it. Ultimately I went with an Ubiquiti ERL, mostly because I didn't really want to buy an RS-232 cable, but the PC Engines boards are probably one of the best fully-DIY options you can get.

Interesting. I have been using such PCs in my Chinese office for about a year. They are made in a factory about 30 minutes from me. I was considering designing a better looking case and bundling a more reliable power supply to export these but I got busy with bigger business.

These Shenzhen factories are somehow getting these Intel CPUs for next to nothing. Factory price for the i5 model was about $100.

At least in the past, I've heard of people getting lots of low binned / questionable QC'd intel chips for dollars per chip. They basically go for auction to the highest bidder. If you're willing to deal with a high defect rate (Either extensive QC yourself or just don't give a shit), it is a pretty good deal.

Could you please recommend accompanying parts if someone wanted to use that board with pfsense to end up with a home/office wireless router? I was looking for a packaged solution but the official pfsense two port appliance with the wireless option seemed fairly expensive.

Sure. I ordered:

APU2C4 http://pcengines.ch/apu2c4.htm

case1d2blku http://pcengines.ch/case1d2blku.htm (black cases run cooler)

http://pcengines.ch/ac12vus2.htm (power plug)

db9cab1 http://pcengines.ch/db9cab1.htm (serial null modem)

I already had a USB->Serial adapter. For storage I used a 32GB Class 10 microSD card in a SD adapter, which I found acceptably fast (and faster than my usb sticks.) Note, if you want to run Snort or Squid, you might want a small SSD, but for a router-only it is overkill.

I assembled the board+case, wrote the 4gb nano-bsd image of pfSense to the SD card, and booted right up.

Total cost was < $200 USD.

edit: I noticed you mentioned wireless...I've heard the wireless support in pfSense using PC Engines boards is better than it was, but I prefer to use (multiple) separate APs for wifi. One of the APs I'm using is: http://www.amazon.com/TP-LINK-TL-WA801ND-Wireless-300Mbps-Re..., which I've had no issues with. I prefer the ability to upgrade/replace my APs without messing with the router, and my router is in the basement, while the APs are on other floors.

I'll probably move to 5GHz wireless later this year, and this separation allows that.

Awesome, thank you!

Seems to be missing on of the most crucial parts: Keeping the software up to date to avoid being a victim to security issues.

During the "Linux-Setup" part of the article:

""" ...and whether you want automatic security upgrades. (Spoiler: Yes, you do.) """

This is also a nice simple and cheap device running OpenWRT ($25) with Wireless N, 2 100 mbit lan and USB: https://revspace.nl/GL-iNet - http://www.gl-inet.com/.

I've got a about 50 deployed, managing them with Ansible, super nice and cheap. USB powered as well.

If you like that, check out the Nexx WT3020H. Very similar specs but you can get them from China for about $13 USD.

Best of all, they're based around a MediaTek CPU, which doesn't have the same USB quirks as the Atheros AR9330 used in the GL-iNet.

I've personally upgraded my 3020H units from 8MB SPI to 16MB, but I've also heard that you can order them directly from the factory with 16MB if your order is large enough, or they're willing to customize.

At least in my setup, a small SSD means that a complete reboot for the router takes 22-24 seconds, and so TCP sessions will not drop.

Be careful when using a very small SSD or something like a CF card for a router. Enabling logging to disk can wear the flash memory out in a matter of weeks

I have a 64GB SSD and a little over 4.5GB used in the last year. I don't think that flash exhaustion is likely in the near future... but I also have a 32GB USB stick plugged in, which gets a backup copy once a week. The great thing about building your own router is that you can easily replace parts.

I have been using a non-commercial, 8GB CompactFlash card with my pfSense install for 2 years with no problems, and I run the full pfSense rather than the NanoBSD version.

I do not run extra services like Squid or Snort, so most of my "writes" are probably within my massive 1GB of RAM.

With modern SSD that isn't going to be a concern. You can write to them at full speed constantly for months and real world tests have shown them to last just fine and you are not going to max out 500MB/s writes to an SSD with log files :)

1400 - 2400 usd for that small box? Is that worth it??

It was $250 before, a lot of sellers on aliexpress just jack the price when they're out of stock instead of delisting the item.

It is almost 4 years since World IPv6 Launch. I’m very disappointed that, other than a few randomly timed rants from Iljitsch van Beijnum, Ars Technica has made no visible movement to IPv6. No AAAA record for Arstechnica.com, no guides to installing IPv6, and now a tutorial for setting up routers spreading FUD about how difficult it is to install IPv6.

I have been running Linux boxes for 20 years as my home router, but just recently bought a Cisco RV325. Sort of got tired of maintaining it, and it took allot more power.

How will these smaller, embedded motherboards handle 1G Ethernet? Will be getting google fiber within next year.

Been using an APU board since they came out and I have a 1Gbps fibre connection. Unfortunately my measured speed comes to about 500/700Mbit but I belive that's due to either shitty equipment in the city wide fibre grid or my own switches/cables. I'm not really a network tech.

Either way the APU handles it fine for a home network and generates no noticeable heat.

Curious if the 120 GB SSD could be doubled as network storage acessible through WiFi?

Yep. Just install the samba/nfs/iscsi/other networking daemon and off you go. About the only thing to keep in mind is that you are going to need to configure the daemon before you use it to ensure that it's not listening on your WAN address, because you probably don't wanna have your files visible to all and sundry.

I'd like to play with Linux on this multi-nic board when I get some time and money http://www.banana-pi.org/r1.html

I use MikroTik for the nice hardware, low power, and RouterOS.

I bought MikroTik for the netflow feature, which can reveal active malware via hardware packet counters by endpoint.[0] The next cheapest router with netflow is in the thousands of dollars. MikroTik is $180 at Amazon.

[0] http://www.irongeek.com/i.php?page=videos/houseccon2015/t302...

Which one do you have? Mine is an RB751U-2HnD and it is plagued with problems, to the point that I set-up an automatic reboot every other day and I'm thinking about switching to Ubiquiti gear...

A client of mine has a RouterBOARD 1100 X2 AH that I also decided to reboot daily for stability reasons. A RouterBOARD 951G 2HnD used in a branch office lost all IPSec configuration and hangs when viewing the settings on the command line. Certified MikroTik technician says the thing has to be reformatted and set up from scratch.

My first experience with MikroTik products. Not good.

Just to chime in with another anecdote, I've got several MikroTik RB493G's acting as PPPoE access concentrators that are stable as hell and have been in service for a couple of years with no issues.

In general, I've had better luck with the lower end of their product line.

I bought this one:

MikroTik - CRS125-24G-1S-RM

http://www.amazon.com/MikroTik-CRS125-24G-1S-RM-rackmount-en...

I haven't pushed it much, as RouterOS is very powerful but has rather a steep learning curve. I've never had to reboot it in the five months I've had it.

Try running 6.32.4 on that unit.

Looks like he hasn't hit the ip_conntrack_tcp_be_liberal problem/setting yet. Good luck with streaming Netflix with that router...

Care to explain?

Here we go: https://www.pitt-pladdy.com/blog/_20091125-185551_0000_Linux...

Anything recent? That is from 2009.

ip_conntrack_tcp_be_liberal is still in kernel sources and it is enabled in distributions like openwrt. The author does not mention this, so very likely that his/her custom router will drop traffic. Recently I ran into the issue with Netflix traffic which seemed to use window scaling. In my case I did not disable scaling, but had to enable this option on arch Linux.

I have a lot of respect for those who know iptables well enough to make things like this. It looks so fascinating, but so complex.

Me too! As others in the thread have mentioned, OpenBSD and pf make for a (IMHO) much easier configuration. Not sure what kind of difference in performance one might expect. I suspect both Linux and OpenBSD are more than capable of keeping up with any traffic one might throw at such a router.

BSD is at a huge disadvantage in QoS capability, to the point that it really shouldn't be recommended as the OS for the gateway on a typical low-speed bufferbloated residential ADSL or DOCSIS connection.

I don't see any disadvantages of BSD for QoS.

I believe the syntax for writing QoS on FreeBSD and OpenBSD provides very good expressive capability [1]. By using tagging [2], one can assign QoS priority to anything that a firewall rule can define.

Having used FreeBSD QoS on dial-up, ISDN, DSL and cable over the years, it is this expressiveness that is one of the reasons I prefer the pf packet filter and thus BSD.

Here's an example for bandwidth limited wan. Interactive ssh sessions get a queue with a minimum bandwidth; scp and sftp bulk transfers go to a separate queue.

  queue rootq on em0 bandwidth 100M max 100M 
  queue ssh parent rootq bandwidth 20M 
  queue  ssh_interactive parent ssh bandwidth 10M min 5M 
  queue  ssh_bulk parent ssh bandwidth 10M 
  queue std parent rootq bandwidth 20M default

  block return out on em0 inet all set queue std 
  pass out on em0 inet proto tcp from any to any port 22 set queue(ssh_bulk, ssh_interactive)

[1] PF - Packet Queueing and Prioritization http://www.openbsd.org/faq/pf/queueing.html

[2] PF - Packet Tagging (Policy Filtering) http://www.openbsd.org/faq/pf/tagging.html

What's the actual queuing algorithm in use? My experience with BSD QoS has been that it's easy to set up, but far inferior to the more modern QoS options in Linux (fq_codel, etc).

I just wish tc had an interface understandable by your average sysadmin. I have to dig into the source to figure out what anything does, but not many users will have the ability/desire to do that.

OpenBSD doesn't support any form of Active Queue Management or fair queuing. It used to support an old and mostly useless AQM (RED) but dropped support in 5.6. Now there isn't even a way to apply ECN marking.

A hierarchy of dumb FIFO queues does not make a real QoS system. It can produce reasonable-looking benchmark numbers when the prioritization rules and benchmark are contrived to match, but in the face of real-world traffic that uses HTTP to carry vastly different kinds of traffic over links that don't have constant bandwidth and latency, OpenBSD is hopeless. Even the rate-limiting capability that OpenBSD has is rudimentary and lacks the ability to account for per-packet overhead and framing overhead, which is necessary for accurate traffic shaping on common service types like ADSL.

If you say you don't see any disadvantages for OpenBSD on QoS, then your idea of QoS is twenty years out of date.

FreeBSD and PFSense do currently have ECN. I should have limited my comments to them.

You're right, OpenBSD no longer has ECN, and my comments regarding OpenBSD were out of date.

And yes, ECN is far more effective because it provides feedback to the sender.

Yes, they've got ECN capability, if you're satisfied with RED as your AQM. pfSense apparently also has an implementation of plain CoDel. Work is also underway to bring fair queuing to the BSD world so that networks with more than one simultaneous traffic flow can get some good QoS: http://caia.swin.edu.au/freebsd/aqm/

All of the BSDs are still way behind; the best by only a few years, while others are stuck in the 1990s.

I believe you can use ECN with CoDel. None of this is restricted to a single flow.

CoDel doesn't do fair queuing so its performance on a mixed stream of traffic is much worse than fq_codel and other AQMs that have a fair queuing component. Please, read some of the literature on what's been developed in recent years. You don't seem to know anything about this subject other than having seen some of these terms show up in pfSense configuration pages and thinking it means it's at parity with Linux.

Perhaps it would be helpful for you to cite some of this literature.

I doubt it would help to point you to anything specific, since you've managed to not find for yourself any of the wealth of relevant information that's been written over the past several years, from draft RFCs to LWN articles and Wikipedia articles to blog and mailing list discussions by internet luminaries and even some of their comments in recent HN discussions. At some point you have to actually put in the minimal effort of googling a topic and skimming at least one of the results, instead of assuming that the people around you are making stupid and obvious mistakes.

Going back to the first article: it was around $300. But as a project for my home, the lack of wifi is more frustrating.

But this article is about building a router, not sure what wifi has to do with it.

I guess most people, when they hear the word Router, they think of their DSL-box. And those commonly have a WiFi access-point built in. (commonly also called CPE (customer premises equipment) by the telephone companies).

    $ ip route
    default via x.x.x.1 dev eno1  proto dhcp  src x.x.x.x  metric 1024 
    x.x.x.x/nn dev eno0  proto kernel  scope link  src x.x.x.x
    192.168.2.0/24 dev br0  proto kernel  scope link  src 192.168.2.114 
    192.168.2.1 dev br0  proto dhcp  scope link  src 192.168.2.114  metric 1024

    $ ip link
    (...)
    2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> (...)

Network guys, when hearing the word Router think of something that holds a full BGP view...

    BGP router identifier 213.200.87.253, local AS number 65534
    BGP table version is 7863026, main routing table version 7863026
    578096 network entries using 58387696 bytes of memory
    578096 path entries using 27748608 bytes of memory
    344077 BGP path attribute entries using 20645160 bytes of memory
    129723 BGP AS-PATH entries using 3908750 bytes of memory
    1054 BGP community entries using 58126 bytes of memory
    4 BGP extended community entries using 96 bytes of memory
    0 BGP route-map cache entries using 0 bytes of memory
    0 BGP filter-list cache entries using 0 bytes of memory
    BGP using 110748436 total bytes of memory
    Dampening enabled. 145 history paths, 277 dampened paths
    BGP activity 730922/149406 prefixes, 731552/150036 paths, scan interval 60 secs
    
    Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
    213.200.64.93   4  3257 2823796   74946  7863002    0    0 3w5d       577951

Woodworkers, when hearing the word Router think of something that can effortlessly carve a groove into wood.

So, who's right?

Well, all of them. A router in computing parlance is just any device that forwards packets between networks. So the first two examples are both routers.

I'm disappointed that you didn't show some text for your last example. I'll make a little marble run for you.

    __________
    ________  \
            \  \
             \  \
              | |
              | |
              \  \___
               \_____

Thanks, your help in visualizing woodwork is really appreciated ;-).

Old network guys remember when networks were not all IP, and there were Novell IPX routers, and AppleTalk routers, and DECnet Phase IV routers, and CLNP lived on for a while due to phone carriers and IS-IS. And even in IP, BGP is only the latest in a long, sad story of routing protocols, and other protocols are usually used inside networks… including IS-IS.

IS-IS (in a somewhat variant form) lives on in 802.1aq.

Why not just have GTT send you a default route if you're gonna run BGP at all?

Oops. While I know that wifi access points and routers can be separate, I'm used to people conversationally combining the two as "routers." Sorry about that!

Not only can they be separate, they should be separate.

Think of where you want to place your router ? (Usually in your server closet or close to where the fiber enters your house) Now think of where you want your access-point to be. Preferably somewhere central in your house, where it will give you the best coverage.

So the best place to put an AP is usually a shitty place to put a router and vice-versa. They have no business being in the same box.

realistically, what is the best option for adding wireless networking to this or any other setup based on a generic box? i assume one can build an access point with a typical wifi dongle but i am not aware of any of the software means required.

On Linux you can use hostapd and dhcpd.

Realistically, WiFi driver code sucks, and there is inherent tension between routing, which benefits from wires, and WiFi, which benefits from open space. See, for example, how the Google OnHubs have the minimum of just 1 LAN port.

You can add WiFi to a generic box, but the Ars Technica staff are promoting a consensus that it’s better to use a separate access point that is designed to be good as an access point.

http://arstechnica.com/gadgets/2015/10/review-ubiquiti-unifi...

Wtf. When I open the link or search alibaba, all of these mini computers is listed at $1300+ :S

Reply on news.ycombinator.com