Hacker News Reader

Comment by secure | Hacker News Reader

Author here. Happy to answer any questions you might have :)

Why the external rebooter? Do you have stability issues with the APU? I have a couple running in production and never ever had to reboot/reset them... Also, don't they have hardware watchdogs?

The apu does have a hardware watchdog, and router7 uses it.

I don’t see any stability issues with the apu at all.

The external rebooter isn’t used as part of a watchdog setup. Instead, it is programmatically used by rtr7-recovery, with which you can recover from a faulty update, to trigger a PXE boot. This way, you can update your router each day in an unattended fashion, with automated rollback in case connectivity is lost with the new software.

it'd be cool if there was an HA mode for this so you can have two devices - and it rotates the active device while the secondary is updating... and then it'd be easy to have power redundancy etc.. .

I can see how that might be appealing in an enterprise setup, but I have no desire to build that complexity into router7 :). In case my apu breaks, I have another one and can just restore today’s backup onto that within 1 minute.

Would it be a lot of work to have a parallel partition which is updated and swaped? If something goes wrong you simply boot the old partition. Like on new Android devices : https://source.android.com/devices/tech/ota/ab/

gokrazy.org, and, by extension, router7, already does this :)

It would be interesting to add cake sqm as that is the only reason i'd currently prefer openwrt/lede over this. Cake should be in mainline linux soon. (https://www.phoronix.com/scan.php?page=news_item&px=CAKE-Qdi...)

How would one go about implementing this? Call the tc binary or reimplement tc in go?

You can prototype/test the feature by calling tc (interactively via https://github.com/gokrazy/breakglass). Then, a good long-term solution is to reimplement what tc is doing in Go.

I've always wondered, how does this compare to a dedicated hardware router (some kind of ASIC maybe. I'm not sure if what routers on the market use), in terms of performance. E.g. if the router is connected to 200 clients and they all experience heavy traffic, what kind of CPU/mem is required to keep up with the traffic?

Wow. I understand now where you were going with google/nftables. Awesome work !

Do you know if it's possible to pack arbitrary files in a gokrazy image ? Like configuration files, data files, or any other native binary ?

Thanks!

gokrazy images currently only contain Go packages, so depending on what you want to do, the best way might be to bundle the file(s) into a Go package.

For router7 specifically, rtr7-recover’s -backup flag accepts a tar.gz archive, which will be unpacked to the persistent data partition /perm. This mechanism is used to restore state and configuration when reverting faulty updates. You can use this mechanism, or https://github.com/gokrazy/breakglass if you prefer an interactive shell, to place files in /perm.

That said, I was thinking about an -overlay flag for the gokr-packer, too, so that you could indeed place additional files in the static root file system. I’ll see if it could be done any other way, but this might be a good escape hatch for when the pure-Go model isn’t applicable (e.g. large legacy applications which cannot easily be ported to Go).

This is pretty awesome. Have you done load (pps) tests to see how efficient it is vs. other stuff?

(Oh, I guess linux is handling all the actual routing, of course. So really the question is performance of dhcp/dns/etc. at peak on the hardware, vs. alternatives.)

Thanks! The router setup achieves the full Gigabit on my line, both on IPv4 and IPv6.

Regarding DHCP, DNS and router solicitations: don’t worry about it. In my network I get barely 1 request per minute (with 31 devices). With the 1 GHz quad-core in the apu2, many orders of magnitude more traffic should easily be in the cards ;)

> Thanks! The router setup achieves the full Gigabit on my line, both on IPv4 and IPv6.

This only requires [1,000,000,000 b/s / (1,538 B * 8 b/B)] == 81,274 f/s for full-sized (1500 byte) frames.

The 'fun' starts as the frame size drops. At the far other end, for minimum-sized frames [1,000,000,000 b/s / (84 B * 8 b/B)] == 1,488,096 f/s.

For an APU2, the first is relatively easy, and the second, nearly impossible with kernel-based networking.

Where would a person even begin to learn how to do something like this? I’m a programmer but wouldn’t even know where to start on a project like this.

There are a bunch of tutorials out there on how to build your own router based on Linux or one of the BSDs. I’d recommend such a tutorial as a good starting point for a top-down view.

The lower-level is very visible through Wireshark, with which you can capture all network traffic.

Does that help?

Most universities with computer science have courses in networking, which usually includes programming a router. I’m sure some courses are available online

Also search for tap & tun based tutorials. Make sure to be ready to read tons of RFCs in the area of neighbor discovery, path finding in graph , and parsing

Why did you not implement it on your Turris Omnia? If the only problem was software update, you still have great HW platform.

I considered the Turris Omnia, but ultimately decided in favor of the apu2c4. Here’s the comparison I made back then:

Turris Omnia: 209 € (-), built-in SFP (+), UART (-), GOARCH=arm (-)

apu2: 105 € (+), no SFP (-), DB9 serial (+), GOARCH=amd64 (+)

Basically, the apu2 costs half (I now have 3), has a real serial port (now permanently attached to my workstation) and an architecture (amd64) which is closer to what gokrazy already supports (arm64) and more likely to be useful to others — I could find way more suitable (≥ 2 ethernet ports) amd64 boards/mini-PCs than 32-bit arm boards.

Hope that makes sense!

Edit: I do agree that the Turris Omnia is a pretty good platform, though! See https://michael.stapelberg.de/posts/2017-03-25-turris-omnia/ for more thoughts about the device.

The apu2 has all the Omnia has (aside from the built-in SFP), and is well-supported by PC Engines and their vendors. They run coreboot and even sell you recovery SPI flash chips for a few bucks if you want to change coreboot yourself :).

Would it be difficult to port this to other hardware? I would assume there is some HAL? (genuinly curious, have no experience with router sw/fw)

The HAL is Linux in this case, but you might need to send pull requests to enable additional drivers, depending on the hardware.

The currently supported platforms are the Raspberry Pi 3 B, Raspberry Pi 3 B+, the PC Engines apu2c4, and qemu x86-64 (I’ll update gokrazy.org with an overview about this in a minute). As long as your hardware is similar enough to one of these, chances are things just work.

Edit: https://gokrazy.org/platforms.html now describes the support level of the various targets.

I guess that has to do with firmware and schematics not always being available. Current consumer routers often have no or little documentation on the internal workings and some have started to limit functionality if you can't use vendor-signed firmware. On the other hand, as soon as you can run stock linux kernels with reasonable performance, gokrazy becomes possible again.

Edit: I was wrong, the Turris has docs and firmware to play with.

Have you talked to init7 about resolving this issue on their side? What is the route of the problem?

I have. They are currently deploying a new DHCP platform and want to spend their efforts on that rollout instead of delaying it any longer by debugging/changing the existing infrastructure.

How much time did it took ? And what NEW technology you need to learn to achieve this ?

I decided to work on the idea at 2018-05-18.

The first milestone of obtaining a DHCPv4 and DHCPv6 lease from my ISP and successfully pinging google from my laptop through router7 was reached on 2018-06-02.

I switched my home network to run on router7 on 2018-06-15.

Today is 2018-07-14, so it took about a month to polish before actually being able to publish it. But it was good enough to use less than a month after starting work on it. I only have a little bit of time in the mornings and evenings before/after my day job, and on some weekends, i.e. it could have been done in less time when working on it full-time.

I learnt about PXE booting, MBR boot loader code, details about DHCPv4, DHCPv6 (had previously not looked at DHCPv6 at all), how wireshark does ssh remote capture, and a bunch of other things I’m blanking on right now :). Lots of interesting details to be learnt in this space!

I find this single comment alone quite inspiring, let alone the project itself and its scope. You took a complex project from idea to working prototype in less than a month, while keeping the discipline and interest high enough to work the planning and implementation around your dayjob. And all the while, you clearly had enough knowledge to know what you didn't know, and enough planning to learn enough of it to push the project forward.

Meanwhile, some programmers (myself included) spend months dithering over the decision whether or not to start that simple web project that's been hanging around in their head for a year. Lots of respect to you. I'll be coming back to this comment again and again.

Thanks for the praise, and I’m glad if the comment helps you in any way

You are really good on discipline.

wouldn't it have been easier to just install dhcpcd?

When considering the total time spent, yes.

However, I had previously run dnsmasq instead of the default kresd so that the Omnia would resolve hostnames in my local network from DHCP leases, and it was such a pain! Updates would frequently break the dnsmasq setup, so I was glad when they finally integrated DHCP lease-based name resolution in the Omnia.

For me, running custom software on the Omnia is a frustrating experience. Working on router7 was a pleasant experience.

Lastly, from my experience with gokrazy.org, I expect the total maintenance/debugging time for the router to be really low, and not in C and obscure build systems, but in Go, about which you can read my thoughts at https://michael.stapelberg.de/posts/2017-08-19-golang_favori... :)

I struggled with the auto updates on the Omnia as well. I wish they'd upstreamed their code so I could run vanilla OpenWRT.

I'd say the Turris Omnia will slowly die in favour of the Mox [1] so it's probably a good time to jump.

1: https://mox.turris.cz/en/overview/

> Updates would frequently break the dnsmasq setup

That's surprising to hear. Dnsmasq has been my go-to solution for almost 10 years. Dhcp hostnames via DNS just works out of the box. I have two subnets at home, serving dhcp 4/6 on both. The only thing that was a pain (mostly because I had to understand it first) was setting up dhclient6 for prefix delegation and get hooks in place so the dnsmasq config would be updated when the prefix changes, but even that addition already survived a dist-upgrade already.

But should that stuff crash and burn next time I dist-upgrade I might take a look at router7, even though I personally really dislike go, but along as I don't have to write code in it I'm fine. Considering how happy I'm with i3 I'm pretty sure you did a good job here too. :)

I’m not saying that dnsmasq itself broke (though I have seen my fair share of weird effects with dnsmasq itself, too), but the _dnsmasq setup_. On the Omnia, you need to somehow get the default kresd resolver out of the way, or make it forward queries to dnsmasq. Either option was frequently overwritten by an update, effectively breaking the setup.

And thanks for the praise, but if Go is not your thing, maybe this is not the best project for you :).

Reply on news.ycombinator.com