Printf Oriented Message Protocol
github.com
github.com
- Binary message formatting for efficiency and data integrity (float to/from string is messy)
- Familiar printf()-style format strings that the compiler can check[*] gives type safety!
- No preprocessing and/or code generation from message templates or anything like that.
Really nice!
Edits: markdown and grammar.
[*]: This uses an extension I think to mark the argument as a format string
As someone who is not the author of this blog post, I'm glad you are impressed: I wrote pretty much the same thing around 7 years ago: https://www.lelanthran.com/chap2/content.html
The difference with mine is that I intended for the library to be endian-independent, so usable over a network transmission to a different endian machine.[1]
Great minds ... (everyone knows how the rest of it goes).
[1] I also eschewed using `%` for the format string specifiers, because (in my words at the time):
> Our parameterised functions must take a specification that tells it how each field should be written, very similar to the printf and scanf family of functions. While it is indeed possible to reuse the standard format specifiers as our own field specifiers it might not be a good idea to do so as this would break The Principle of Least Astonishment.1 This is because anyone who is changing the code and who sees a string literal with well-known format specifiers such as %02x and %c would naturally (but incorrectly) assume that all format specifiers are supported. We do not wish to confuse the reader.
JSON and protobuf make that easy. You can start emitting the new field and then update the receiving end at your leisure.
With these libraries you have to write your own versioning system. For a small performance improvement over JSON parsing it's pretty much not worth it. There's almost always going to be a piece of lower hanging fruit.
For example, imagine you are sending message type 1 with format "%d %d" and later you realize you actually need three instead of two ints. You introduce message type 2 with format "%d %d %d" and update your readers to support both types. Once those are deployed in production, you update your writers to send the new message.
This is kind of the opposite of what happens with protobufs, where unsupported fields are silently dropped by the receiver, so you can update the sender before the receiver. But this is arguable less safe, since if the receiver drops some fields, it might not interpret it the way the sender intended. In that case, it might be more sane if the receiver outright refuses to process a message it doesn't understand.
Source: I worked on Parrot's code. And you can make your mind by looking at the other projects in Parrot dev group...
Writing good code is not impossible. Or even difficult.
It just requires engineers to hold themselves to a higher standard.
For starters, engineers will disagree on what "a higher standard" means.
(Indeed, I've worked in industry sub-sectors where some of these best practices are widely understood to be counterproductive. Videogames often do few if any unit tests, preferring instead to rely on an army of human testers reviewing every build over and over because you can't generally capture the je ne sais quoi that is the feeling of "fun" in a test or even a design doc. By the time they've rapid-developed until they have a prototype that feels right, that prototype is the product and it doesn't make sense to add tests at that point because you can just test that one artifact to death by hand and ship it. This does, indeed, damage the reliability of the end product and result in something that often needs a week or two of follow-up patches... But because for all they complain, gamers don't actually stop buying games day 1 because of the bugs, the industry as learned leaving them in is acceptable risk that saves time).
If we shifted resources to supporting maintenance over novelty, this could change... But the people with the money want to make more money, and that mostly happens in the "come out with something novel and everyone gives you a buck" space, not the "keep the electrical grid control architecture running" space.
Incentives drive the industry.
They don't force you to share the format strings at connection time. In fact, it is just a thin layer around sockets and you just override the message handler. I like it! Might be improved further with something like libfmt which is also used by spdlog.
In the API, the string length is determined the normal way in each language: strlen() in C, len() in Python. The trailing NUL allows the C decoder to return just a pointer to the string (without allocating).
So internal null bytes and strings longer than 0xFFFF are functionally prohibited.
This library takes the idea of modern type safety and throws it away. Instead, the library leans in on `printf()` instead, which is known to be unsafe. And it does it in a memory-unsafe language.
...on a drone. Where safety needs to be important (even a small drone can do significant damage).
It's neat and all. But that's a killer anti-feature in my opinion. I wouldn't use it on my drones.
The reason why I missed it the first time is because this information is introduced in the second half of a paragraph, where the first half is about a different topic. This is a bad way to divide paragraphs.
> The encoding/decoding is done with printf/scanf like functions with a format string and a variable number of arguments. However, no actual string formatting is done, the payload is a binary representation of arguments.
Seems to encapsulate the complexity of having a string formatting interface to a binary protocol reasonably clearly, but is not crafted to withstand speed reading or skipping over sentences.
- bindings to other languages: I'm seeing C/C++ and Python supported. I could, hypothetically, use anything with a C-native interface to wrap that C library and bind to other languages, but that's work I'll have to do. And that binding is going to be as easy-to-use as a wrapper around a C-native library.
- versioning (or some other way to deal with drift between serialization and deserialization). In practice, protocol buffers are often used to do both long-term storage serialization and short-term on-wire transmission. Both use cases always eventually involve two services that cannot be kept in lockstep sync on releases (either because you have to read older files or because you needed to release your backend before your frontend, do a rolling release, whatever).
This is simpler than protobuf, which is cool (and I'd recommend looking over its implementation to figure out how to write something like this), but if you put it into implementation in a non-toy distributed-system project it will not scale. It may have other applications though; I can definitely see it as the backbone for a simple videogame messaging protocol.
Still, no harm in writing a thing and putting it out there; I'm happy it exists even if I won't use it.
Plenty of distributed systems don't have the requirements you outline or need to scale. Distributed systems are not necessarily large, cross organizational lines, or require rolling releases. Sometimes the tradeoff in complexity by, for example, introducing a dependency on protoc et. al., is not worth it. I've built several systems that this would've been a perfectly fine choice for.
I'd also point out that because it's so simple, adding implementations in new languages should be fairly straightforward, but I think this really shines in C codebases where complexity must be mostly dealt with by hand (for better or worse).
(But fwiw, if you did want to inspect this traffic as human-readable, it would be insanely easy to create either a packet sniffer for this or add support to tcpdump/wireguard. It's just dumping a printf format code and some raw data onto the wire, so you just pick up a new frame, read the format code, snip the data from the frame, pass it to printf() with the format code and the data. It's like 12 lines of code, no library needed other than libc. As a hacker I find this way more exciting than using a bloated standard to pass utf-8 strings)
And code gen and protocol descriptions bring a certain useful discipline to modifying existing protocols.
But it has that funky C style so I bet people using C like it.