Ubus (OpenWrt micro bus architecture)
openwrt.org
openwrt.org
It was circa 2009, I had only been exposed to plain text protocols and didn’t know about json. In hindsight, we might’ve been better using standard dbus, of protobuf, but I was a rookie and it provided the performance we needed (for DTV metadata).
I’m happy to see that these can still thrive, and I just recently figured out that discovery is a net multiplier in these projects; doing this really proved me that any problem is solvable if you can have some time to think about it and can prototype.
I long for these moments now, I feel like nearly all computing issues have been solved and we are now just plumbers, connecting libraries and software modules through config files instead of building things.
Also I think "any problem is solvable" needs some qualification - there are a ton of problems that have yet to be solved or are super complicated. I still haven't figured out how unbounded model checking works for example.
It's all a big pile of stateful daemons notifying other daemons with a billion race conditions and zero debugging capabilities, like a parody of how not to create reliable systems. When you have shell scripts parsing JSON messages, you know it's over.
Newline-delimited != SOTA
https://en.wikipedia.org/wiki/Apache_Arrow :
> Arrow allows for zero-copy reads and fast data access and interchange without serialization overhead between these languages and systems
JSON lines formatted messages can be UTF-8 JSON-LD: https://jsonlines.org/
Linux networking is now all built on eBPF.
> In general when we write to a socket, or a file, or in our case a pipe, we’re first writing to a buffer somewhere in the kernel, and then let the kernel do its work. In the case of pipes, the pipe is a series of buffers in the kernel. All this copying is undesirable if we’re in the business of performance.
> Luckily, Linux includes system calls to speed things up when we want to move data to and from pipes, without copying. Specifically:
> - splice moves data from a pipe to a file descriptor, and vice-versa.
> - vmsplice moves data from user memory into a pipe.
> Crucially, both operations work without copying anything.
But then to scale to more than one node, everything has to be copied from the pipe buffer to the network socket buffer; unless splice()'ing from a pipe to a socket is Zero-copy like sendfile() (which doesn't work with pipes IIRC).
"SOCKMAP - TCP splicing of the future" (2019) https://blog.cloudflare.com/sockmap-tcp-splicing-of-the-futu...
[1] https://web.archive.org/web/20131209010702/http://unixbus.or...
iw event | awk '/new station/ {print $4}' | xargs -n 1 sh -c 'ubus send new_station {\"mac\":\"$1\"}' _
(Yes, this contains silly hacks to work around busybox limitations)
I don't know if we now have a better alternative that is similar in speed, ram use and binary size.
If we do I'd would read a discussion on which one to use with interest.
edit: Also D in safe mode like mentioend above - not sure if it was around in 2010?
On some devices you have 32MB flash storage in total.
32 Megabytes, not gigabytes. This is not a typo.
This needs to fit the bootloader, the Linux kernel, the initrd, the rootfs containing all the user-space tools making OpenWRT an actual usable OS and whatever daemons you need to implement your particular network needs. Oh and you probably want the WebUI too.
In 32MBs. There are newspapers online which loads more than that just to show the front page!
There’s no room for 2MB HelloWorld type languages in this space.
Eg on https://ada.godbolt.org/ you can see the default program compiled to ~25 lines of assembler.
edit: actually it contains a call to a bounds check function that presumably is in the runtime library, i don't know how big that is.
A major point in this is that C also has a fairly big runtime library that should be factored into the comparison!
But libc will be there anyways, unless you are rewriting absolutely all of userland.
The original releases of OpenWRT used uClibc, which is nowhere near Glibc levels of bloat (Musl beats it on code quality and is used today, but didn’t exist back then). Also, yeah, you’re going to have a libc on a Linux system no matter what, so this is one of the rare cases where dynamic linking makes for a legitimate optimization.
You can still build software for OpenWRT that requires the much bigger Glibc, but of course it will not work that well on devices with limited memory.
464.7K Oct 15 2022 /lib/libc.so
20.0K Oct 13 2022 /usr/lib/libudev.so.1
8.0K Dec 15 2022 /sbin/udevtriggerYou have to consider the surrounding ecosystem. Those interested in such languages are not necessarily those interested in contributing solutions to the problem space. Any project attempting to use such a language in OpenWrt would very likely not have survived until today.
You can write C code such that its unit tested, fuzzed, statically analyzed and reviewed, and often embedded code has one or two specific jobs.
I would have loved to see Zig here, as it makes all of the above (testing, analysis, code clarity for review) easier, but its also not memory safe necessarily.
You could write it in D with @safe (which is SafeD or whatever), which is memory safe, but thats not a very popular language.
You could use Ada, but again, thats not as easy to find devs for as C is.
You could use Go or Ocaml, though go isn't really memory safe and you can easily cause data races in goroutines, and ocaml has a runtime afaik so thats out of the question for low memory devices.
You could use Rust, assuming it had existed back when they started it, and you would get memory safety, but also an entire kitchen sink of useless garbage (like C++). Youd also have to shell out to unsafe{} in a lot of places, unless you use a crate for it, which will then do unsafe{} for you.
So chances are, whatever you do, C is a pretty sane choice, or maybe C++ if you want RAII to at least make your resource management and lifetimes easy to handle.
I dont think you deserve to be downvoted, since this is an interesting discussion to have. However, I think it would have been helpful for the discussion if you had outlined which language youd suggest and why.
To Steelman your argument, I would say you think that C is unsafe to such a degree that even a for loop is UB most of the time (integer overflow as UB), no real way to check array bounds, no real way to catch off-by-one and similarly stupid simple errors, use after free, egc. and that the entire ecosystem relies on raw pointers and macros, and its a shitshow. I think your point would have been to suggest Rust, as it fixes all these issues, while bringing along a stronger type system and a better toolchain.
I'm not saying C is a wrong choice here, but this argument doesn't make much sense to me. All the other languages also come with "useless garbage" (some even with a collector for them). I don't see why you'd need that many unsafe blocks either.
I feel like Rust, as a language, is much better equipped for writing a daemon like this. I'll even go as far as to say that modern C++ (C++2x) is a better choice than C for this stuff with its superior memory and resource management tools.
Of course most of the OpenWRT development is done in C, so doing it in another language requires a very good reason. I wouldn't look at the concept of "a networked daemon exchanging messages between arbitrary services" and think C of all languages is a good fit unless every other developer on the team can only write C.
Zig would've been a nice way to meet halfway, but the language isn't finished yet.
Of course the point is moot because this particular project existed years before any of the modern alternatives or many of the C++ improvements were even available. Still, if they'd start a project like this again, I'd hope they'd pick a better language for it.
Binary size is sometimes a big deal. Two decades ago I investigated using C++ for an "embedded" linux ssh server, but decided the 30kB overhead was too large (target was a 4MB laptop). The server ended up being used in OpenWRT and other places, I'm curious if it would have happened if I'd gone with C++ instead.
I feel like Rust without std might as well be a different language. Almost all of the available mindshare, docs, libraries are dependant on it.
Sure, you can use it. But to me it just looks like technological poverty. Even compared to the C ecosystem.
There were sufficient crates to write a ssh server for a rp2040 (mostly out of curiosity) using what's available in no_std.
That repo has a few crates depending on each other - sunset is the toplevel ssh, sunset-embassy adds no_std async, then the async dir has std rust, with a commandline ssh client in the examples dir.
There's obviously a lot of need for improvement in Rust in regards to small systems, as people have pointed out. But there's also a lot more middle ground in Rust vs, say, C++: You can be no_std and still have Vector, for example, while in C++ the STL is an all-or-nothing thing.
It's a smaller community, and it will come down to individuals deciding it's important and making contributions.
Alas, there is little interest in/awareness of the middle ground between the no_std world and the "size doesn't matter" world in the Rust community right now. I've asked a bunch of people for pointers as to where to start working towards supporting those cases but I mainly got shrugs. There were attempts at more lightweight stdlibs and so on, but they all seem to have fizzled out.
I wonder how much of this is inherent in the language design. Rust heavily leans on monomorphization, so as a first approximation, you will always generate more code (before maybe optimizing that away again). Famously, the Swift people went to great lengths to avoid these problems: https://faultlore.com/blah/swift-abi/. But while running Swift on Linux is possible, this is even more niche.
The key was to turn off panic unwinding, turn on size optimization, and full LTO. I think I recall increasing the inlining threshold helped a bit.
It should be possible to use a shared library for std but it would come with the limitation that the bins must be compiled with exactly the same rustc as std.
In the embedded space, you use small effective languages like assembly and C. You either design for explicit memory use entirely, or you systematically test for memory usage and waste - especially if you're delivering industrial applications. The so-called "memory safe" systems tend to be very expensive for memory usage and space, and not worth the investment. Usually, anyway. This may change in the future but it's a bad change if it comes with additional power usage requirements and the environment also requires minimal power use.