VyOS From Scratch – Edition 1 (set up your own router)
blog.kroy.io
blog.kroy.io
The VyOS website says:
> Democratizing how we access networks through a universal Router and Open source software.
> Our vision at VyOS is to dramatically change how we access networks so that we can all build the solutions we always dreamed of, without restrictions, limitations, or prohibitive costs.
But I'm still kind of none the wiser. Does this thing use Linux or something else?
edit: used to be a maintainer for a short while :)
* You can SSH in and configure it like you would a managed switch or router. There's a single object that models all of the device's configuration, when you commit it a bunch of scripts activate and actaully apply the configuration to the running system.
* Deployments are image based, you can roll back to older images etc.
* You don't need to look at any of the system's underlying configuration files or use any of the normal Linux commands to examine and manipulate the state of the system (the commands are still there for convenience of course). You don't even need to be aware that you're really using a bunch of custom bash functions to examine and manipulate the state of the system.
I personally use regular Debian 12 on my router without problems. It also has "declarative config" since all the configuration, firewall rules, etc are a bunch of config files that I can scp / ansible over any time.
Vyatta is the original OS, based on Debian, dating back to 2005. Its history is detailed here:
https://en.wikipedia.org/wiki/Vyatta
In 2011, Ubiquiti launched their EdgeMax products with EdgeOS which was a fork of Vyatta Core 6.3 ported from x86 to Cavium.
In 2012, Vyatta was acquired by Brocade.
IN 2013, Vyatta Core 6.6 was forked as VyOS.
That's the rough origin of these three OSes.
I used Vyatta Core on a PC at a startup from 2009-2013 as our office router. I haven't paid attention to it or VyOS since then.
I've been running various EdgeOS routers at my home since 2014 or so, first an EdgeRouter Lite and today an EdgeRouter 4.
EdgeOS has been updated quite a bit over the years from its Vyatta Core origins, but the original developers are no longer with Ubiquiti. EdgeOS hasn't seen updates in quite some time now.
Also, not all Ubiquiti run EdgeOS. Only the EdgeRouters do. The rest of their products run a completely different OS, generally either UbiquitiOS or UnifiOS.
Sources besides my own memory:
https://blog.vyos.io/versions-mystery-revealed
https://old.reddit.com/r/Ubiquiti/comments/scqlg3/what_happe...
It's debian plus some shell trickery and CLI tools that let you configure debian and debian packages as a router from one large config tree using neat CLI tools (that support commit/rollback).
Normally you'd need iptables, a separate DNS package, DHCP server, etc etc to set up a router, with VyOS you just change VyOS config and it configures normal debian packages for you.
Plus everything is exhaustively tested and configs are reverse compatible, hiding all breaking changes underneath.
It's super neat and it works perfectly on a £100 fanless Celeron J4125 box from Aliexpress as a home router, routing and shaping 1gbit without breaking a sweat and with deeply sub-ms delay.
Do you have an idea why the CLI tools aren't distributed independently? Why shouldn't I be able to run it on a Debian system I already have (and understand)?
Running an entire new distro just seems like overkill for what it actually does over a normal Linux system. It's just a configuration manager!
As an example, the Ansible modules for VyOS are basicially just variations of an adapted ansible.builtin.shell, instead of offering to manage state in a more first class manner (via attributes and values):
https://docs.ansible.com/ansible/latest/collections/vyos/vyo...
It is not very elegant though.
Do you know of an open source router that does what you are looking for?
To be honest this feels more like a limitation in Ansible, which has always felt like a bit of a hacky config management system to me in that the way it functions is generally to run a bunch of commands that gradually mutate the system's state, rather than atomically applying the target state, but then I've been spoiled by NixOS on my personal infrastructure recently.
https://docs.ansible.com/ansible/latest/collections/vyos/vyo...
https://docs.ansible.com/ansible/latest/collections/vyos/vyo...
https://docs.ansible.com/ansible/latest/collections/vyos/vyo...
Why isn't doing this much more popular? All the systems are already there, after all! Why aren't there (that I know of) dozens of projects to accomplish this relatively easy, but relatively useful task?
I think it's a pretty big deal to be able to configure that stuff from a single place. Commercial router manufacturers all do it. Why does (as far as I know) only VyOS do it on the open source side of things?
Most enterprises prefer buying something with a support contract from a known name vendor (Cisco, Juniper, etc.). Most home users just use what their ISP provides them with, and of those that want something more, they either pick a SOHO vendor like Ubiquiti/Mikrotik, or if DIYing the hardware, choose pfSense / OPNSense / DD-WRT for the clickOps options, because networking really isn't trivial. For those for whom networking is trivial, Debian is fine router OS if you know your way around iptables and friends.
That leaves all those who want to use DIY hardware, and a enterprise-like declarative CLI. That's really not a whole lot of people in the end.
It's a decent-ish option if you need advanced routing functionality; one thing to keep in mind, though, is that unless you're OK with running unstable 'nightly' code, you'll be spending USD 8K+ on an annual basis.
They have an LTS release, no?
They seem to follow the RedHat strategy though, only subscribers can download prebuilt images, but you can build the LTS ones yourself:
Sounds fair to me. Truth is there's no good alternative other than pfSense but if you want Linux (hw support etc) I don't know if you can do better than vyos for routers.
However, if you're comfortable with CLI and modifying configs in /etc/ then just running a bare metal Alpine Linux box is perfectly doable on a tiny box. iptables/nftables for firewall/NAT, dnsmasq/bind9 for dns, dnsmasq/isc-dhcp for DHCP. I've got a handful of these boxes all interlinked via wireguard, sharing routes via BGP using bird.
Sure, you miss the config verification that VyOS provides, but does mean you learn the underlying tools themselves and that knowledge is portable to any other box running those systems.
Personally, I don't quite understand why VyOS is a standalone distro when it could just be a config generator/checker package. Could even support multiple different underlying tools so if you want to use dnsmasq over bind9, or vice versa, it can provide a unified config interface for them.
If wanting internal and external subnets as "zones", iptables/nftables lets you match against incoming and outgoing interfaces. It would be trivial to make match against an incoming interface and jump to a zone specific chain. This is how I manage private subnets. fw-mark is also useful for setting routing rules. Can change which routing table is used by matching rules in iptables.
If wanting to do more stateful things, I'm not aware of any default package, but setting a rule to send packets to an NFQUEUE and implementing some custom logic on that nfqueue would be rather trivial too. I'm sure eBPFs are useable in there somewhere too, but I've very little experience with them.
Obviously iptables/nftables has its own issues, as seen in recent (and not so recent) posts about it being bypassable with raw sockets, but that tends to be host only and not when used as a gateway.
https://support.vyos.io/en/support/solutions/articles/103000...
You create a _zone_. You name it and assign some interfaces to it. For my needs, I only assign 1 interface per zone. Then, you specify with which other zone that zone can receive traffic from. That also comes with the identification of a firewall rulesets to apply to that pair.
So, `'Zone WAN (iface eth0) <- Zone LAN (iface eth1)' => apply fw LAN-TO-WAN`
When you do that, the firewall rules become much simpler to write and maintain.
But, a best practice is to assign every zone to every other zone. This soon becomes a combinatorial nightmare. When you want to add a zone, you have to create 2xN new zone configurations and 2xN new firewall rulesets.
iptables -N eth0toeth1; iptables -P eth0toeth1 DROP; iptables -A FORWARD -i eth0 -o eth1 -j eth0toeth1; iptables -A eth0toeth1 -m tcp -p 80 -j ACCEPT; # add any more rules
Or, as you say to avoid exponential combinations, just make a chain for each zone (interface) and explicitly allow specific protocols/ports to target interfaces. Zones with multiple interfaces are just multiple rules to jump to the same zone chain.
I can't even find anything related to policy-based routing (PBR).
We need something modern - easy clustering, modern API, event stream, gRPC-based plugins, etc. (And yes, I have thought about developing it myself, it's on my pile of TODO)
I tried if few times and every time I stuck on something and messagefrom developers was: this isc-dhcp feature is not supported. This was huge national scale ISP and bypassing those limitations means a lot of $ to adapt surrounding systems providing input to isc-dhcp LDAP DB in its own config style.
or `vbash` in VyOS for all of those :/
But this is my point; why is VyOS a distro when vbash could just be a package available to other distros?
OpenWRT comes to mind, I've been using it for decades on first dedicated hardware, the last 6 years running in a container on a ProxMox box (DL380 G7). It has no problems whatsoever routing at (gigabit) line speed using a few megabytes of RAM and a few cores. Configuration is mostly declarative using UCI although it also offers the freedom (which comes with responsibility) to use scripts. I use the latter to deal with edge cases which lie outside of the purview of normal routing operations, e.g. triggered actions related to the use of Timelimit [1] on my daughter's phone, IoShit things with special needs, etc.
This is must have to be considered by any ISP/enterprise where networking is their core business.
vOS has all three. More or less buggy but they are here.
The above mentioned I feel are good enough for home and SOHO(small businesses).
[1] https://openwrt.org/packages/pkgdata/kmod-mpls
[2] https://openwrt.org/packages/pkgdata/frr
If anyone knows other Opensource routing software that support all of this - let me know. To my knowledge vOS is the only one.
Is there a uptodate reliable guide (possibly including how to persuade your wife it's a good idea to drill holes in the living room ceiling to run cat6)
On the same Celeron J4125/i226 box VyOS was absolutely perfect, not a single issue, significantly low (and always low) latency with higher throughput.
On the hardware side, I think the /r/homelab hivemind doesn't get challenged enough. Dell optiplexes cost very similarly to Aliexpress Protectli alternatives (such as [2]), while being larger, having a fan, and being overall more hassle. TP Link/Ubiquiti WiFi APs seem to be overall inferior to Aruba Instant On, which is exactly the same hardware HP sells in their Aruba line, but for the same SOHO price.
[1] https://teklager.se/en/knowledge-base/opnsense-performance-o...
VyOS finally lets you have all your configuration in one, easily controlled place. Nice!
For some reason I ended not actually trying it out too actively. I think I was weirded out by the distribution model and concerned by the small community.
After using it for several years, the implementation is clearly lacking. It seems that the maintainers are overloaded, because contributions to fix minor issues, or add config options get ignored in my experience. As a result, the configuration is missing some nice options in the IPv6 space (so-called tethering), and it's still using iptables for packet filtering. It's also rather hard to roll your own, with your own modules: the module system is rather hard to use.
Not to mention the lack of interest to roll out a version for ARM.
My next router will be based on NixOS, and will attempt to recreate the awesome UI of VyOS.
Not sure who would want something rolling on device like router.
Nowdays everyone wanting something good and free go OpnSense way.
And I have been unable to build a cloud-init image.
But the ISO is buildable from the LTS branch and it works well.
I'm usually all for up to date software, but on my networking equipment??? I don't really want to beta test that stuff, but that's what they seem to want to make me do.
You can build the ISO from the LTS branch though and that branch doesn't move much. Though, I don't know how you can tell which commit was used to release, say v1.3.2. For the moment, I simply build an LTS ISO using the latest commit of the LTS branch. That strategy has been rock solid for years now.
I need to write a follow-up so bad.
VyOS has evolved a BUNCH since I wrote this, but the same basic ideas apply. Mostly some configuration nodes have moved around.
Do you have any plans to expand into IPv6 functionality?
I use VyOS whenever I need layer 3 routing in vSphere for test environments. NSX-T is (way) faster by dint of being deeply integrated into VMkernel, but VyOS is pretty performant for what it is and is easier to install to boot.
Wondering how much can be automated of the installation/state; would it be possible to use version control? If not, I can see the appeal to suggest Nix over this.
You can also build your own images with docker using the annotated releases.
- up until a couple of weeks ago the hostapd module was basically a toy: could only manage a single SSID, no way to configure the radios, hardcoded to WPA2-PSK;
- the NixOS firewall is still based on iptables and conflicts with nftables, so you must disable and manually write rules;
- the `networking.nat` module (NAT44) doesn't do NAT reflection;
- I had to write a module for Jool (NAT64, SIIT);
- I had to write a module for libreswan (IPsec);
- I had to write a module for automatic rollbacks, otherwise you can loose access if you make a mistake.
Vyatta and VyOS also provide a much higher level abstraction over the software that is being configured (e.g. you don't have to deal with a specific IPsec implementation). Finally, once you do `nixos-rebuild switch` you're on your own, while with vyatta you have a clean command line interface to inspect the state of the router and manage it.
Do you think that, had your use case been simpler, you'd have enjoyed the experience?