Now if they could just get rid of Linux underneath and use something with better security. L4, maybe. After all, this is for embedded devices which basically run one program.
But even if L4 would provide the minimal set of functions expected to pass packets on an embedded device like a router: Would you really gain security if you had to rewrite most of the libraries involved for, say, user-interface and configuration (where all of the exploits reside) from scratch? The whole wpa/wpa2-hostapd madness, or VPN setup with all the many dozens of possible protocol combinations?
And even when one plans to rewrite hostapd, openvpn, dnsmasqd with all their dependencies from scratch, in a safer language than C, with reduced priviledges and isolated from the rest of the system: One still could run these replacements easily under a Linux kernel providing most of the necessary infrastructure and with hardly and bugs being discovered in the kernel core that would be relevant for such an embedded device.
This is a job for a limited protected-mode OS that Just Works. L4, QNX, something like that.
For web attacks, the web UI has to be quite privileged unless you explicitly privilege-separate it, Sandstorm-style. That seems like a big project for a router.
For kernel attacks, SELinux just increases the attack surface. Throw a good seccomp filter at things, deny access to proc and sysfs to things that don't need them, and use hardening options.
The trouble with anything other than Linux or maybe FreeBSD is driver support. And, for a router, even if an attacker merely compromises the network stack instead of compromising the whole system, the attacker still mostly wins.
What would be interesting is a good verified boot or immutable storage model. For example, have the router boot into a mode in which it can't write to persistent storage unless a physical button is used at boot time. eMMC can do this, but I have no idea whether the NOR and NAND chips in most routers can.
The boundaries of what tasks are handled by the CPU vs by dedicated offloads on the SoC vs by the NIC (which usually has software of its own) differs with every manufacturer and every hardware generation. The job we want our routers to do is a moving target as the industry continues to develop new routing and configuration protocols (eg. Homenet) and new QoS techniques and new WiFi rate control techniques that need to be incorporated into the software running on the CPU and/or NIC. The hardware is usually weak enough that the products can only get the job done by prioritizing performance over expensive security measures.
I could get behind the idea of a line-rate dedicated firewall+NAT with formally specified behavior. But any attempt to enumerate the core functionality of a modern wireless router will leave you with a job that is far larger than any successful formally specified/verified software project.
Running atop Linux is the only option that doesn't leave you stranded with a '90s-era feature set and a cripplingly small base of supported hardware, and the userspace stuff is the low-hanging fruit for securing anyway.
Linux is what the chipset manufacturers target, it's what the router manufacturers ship, it's what most of the academics seem to turn to when they're not using a network simulator, and Linux seems to have the most active networking developers. The only compelling argument for BSDs is that pf.conf is more approachable than the Linux tools, but BSD advocates usually don't mention that it's because pf does a lot less than tc and the other Linux tools.
big business usually use their own network stack as stock bsd is at most suited for soho workloads
I think though that the "industry" work on QoS and rate control has been pretty dismal and counter-productive. The major improvements in QoS in OpenWRT didn't come from the router industry, and the router industry hasn't been speedy about incorporating it. Fixes for queuing and rate control in the WiFi stack are also coming from outside (when they come), and who knows how long they will take to get mainstreamed.
But maybe an operating system is too large in scope. You're right that these devices only run one application, why use a tool (operating system) that can run many applications?
It's a bit like running a hypervisor for only one operating system. There are benefits, true.
Sure, they could move all that to BSD, but then if they wanted to do that, why didn't they do that in the first place, rather than working on Linux?