Comment by jchw | Hacker News Reader

Comment by jchw | Hacker News Reader

Parent Full thread

Unfortunately I concur.

There are a lot of problems with D-Bus, but that it wasn't using JSON wasn't one of them.

The tooling for D-Bus is terrible. It has composite types but they are very primitive, e.g. you can't name fields in a structure without extensions (there's like three different extensions for this.) A lot of the code generators can't actually handle real-world schemas.

Now I understand that D-Bus itself is also a problem (as in, the protocol design beyond serialization) but varlink looks like a very solid step backwards. In any single individual use case JSON serialization is unlikely to be a huge bottleneck, but in practice if this will be used for all kinds of things it will add up.

I really wish the Linux desktop would coalesce around something like Cap'n'proto for serialization.

P.S.: It'd probably be possible to shoehorn in multiple formats but I think that's a mistake too. Something like capnp will have a strong impedance mismatch with JSON; mixing formats with different degrees of self-describability is unwise. But then formats like MsgPack and BSON which are direct JSON analogues miss out on the best benefits of using binary serialization formats, which makes them almost the worst of both worlds, and you'd probably be better off with using a highly-optimized JSON library like simdjson.

CBOR is the obvious choice here.

I've suggested it before and the systemd folks didn't seem completely opposed to it. Also because cbor parser is already in the dependencies due to FIDO2 disk unlocking

What is the data rate (messages, bytes) that you expect for this "D-Bus replacement protocol"?

Which fraction of CPU will JSON ser/des will take to justify using CBOR?

I mean conversely every second week on HN someone complains about "inefficient modern development practices" and this is the sort of minor thing which would be trotted out as an example.

So I'd argue it's not an unreasonable question (although I lean closer in: better types then JSON is the problem, since a decent text serialization format is always going to be needed for debugging and development).

This sounds like it wants to be a fundamental IPC facility in Linux, which seems like a major thing.

Any binary format can be dumped to a text when viewed for debugging. That’s really a nonissue. Have the tools dump to JSON if you like.

I genuinely dislike CBOR. Formats which require me to calculate the length of the message before sending it lead to bad library design which often leads to easily compromised code. Add in an "indefinite length" option and you've got potentially unbounded client memory usage to watch out for. As if that wasn't enough you get extensible tags so the meaning of any message is entirely dependent on the context it was sent in.

It gives you a lot of decent protocol wire design and then flatly ignores everything we've learned about these types of designs in the last 3 decades. Be on the lookout for client libraries to slowly add in all of these checks as the vulnerabilities are discovered in them.

FWIW, ASN.1 BER supports an indefinite length encoding (though DER does not; ASN.1 JSON and ASN.1 XML do as well)

ASN suffers from some of the other problems regardless of the encoding.

I was thinking about this more and I think FTP actually had a lot of the right ideas here. You want two channels. One for fast interactive command messaging and a second one coordinate especially just for bulk transfers arranged over the command channel. The design flaw in FTP of putting these on two separate ports put it immediately in conflict with firewall best practices so I think it went mostly unnoticed that it fundamentally is a good arrangement.

What you want is one channel, sequenced packets, with a defined maximum message length that is negotiated at startup and never changes during the life of the channel. This should probably never be more than 65k. There should be a known length packet type, and an unknown length packet type, with any attempt to send more than the negotiated maximum triggering an error and disconnect.

If you do need to send more than the negotiated amount you should open a new connection, in a bulk transfer mode, that after the initial handshake, has no protocol and is merely an associated stream of bytes that are wholly uninterpreted by the middleware layer other than to help you associate it with the sequenced packet that requested its initiation.

You'd actually be able to use TCP (with DSCP even), mostly avoid head of line blocking and multiplexer latencies, and have a reasonable security guarantee in the sequenced packet mode, and never have a protocol which pretends that 4GB strings in the middle of a packet are necessary or even a good idea to "support."

The downfall of this is that it would be much harder to implement it on a serverless architecture and would be nearly as complicated as a protocol as WebSocket ends up being. It might be worth playing with as a concept anyways.

I think the protocol on which PDUs are exchanged on is a bit orthogonal to the PDU protocol itself here, but the split-mode networking protocols (FTP, IRC DCC, mosh, SIP, etc) always do end up getting less used because establishing two different connections is a bigger burden and there really needs to be a great advantage to doing so, and often require other considerations (better connection tracking, TURN/ICE servers, etc).

For a localhost/UNIX domain protocol, it might work since most of these considerations are significantly reduced.

As to DSCP, it might be useful across controlled networks but in general, in my experience it's not really ever honored broadly.

Back on the original topic, when I was building this for my Linux distribution I ended up just using basic JSON with TLS (for client certificate authentication) though since having authentication for remote management was a goal, and once I was already having to perform a TLS negotiation then the PDU consideration for performance wasn't something to really spend too much time on.

You would need a pretty intense QoS system built for this if you are using something like TCP for this using two different connections. Underneath it would still use IPv4/6 packets and you only have so much bandwidth. The bulk transfer channel might stall the transfer of control packets temporarily.

You would need to tightly control and multiplex the messages yourself, needing to do this in one connection or using something like TCP priority flag.

Personally I just think that TCP is a shitty protocol for building applications. In almost every use case either UDP or SCTP are a better choice. With raw UDP datagrams you aren’t guaranteed delivery but it’s great for the kind of telemetry where the last message in is what matters. You can also build quite flexible stuff on top of it.

SCTP gives you in sequence reliable datagrams and congestion control. This means you don’t have to devise message length communication into your application layer protocol and can rely on your transport. It also has multiplexing built right in. If it had a built in checksum it would truly be ideal. Not sure if something like secure communication really belongs at this level but if it had that I double we would ever use anything else. From SCTP’s Wikipedia page:

> SCTP applications submit data for transmission in messages (groups of bytes) to the SCTP transport layer. SCTP places messages and control information into separate chunks (data chunks and control chunks), each identified by a chunk header. The protocol can fragment a message into multiple data chunks, but each data chunk contains data from only one user message. SCTP bundles the chunks into SCTP packets. The SCTP packet, which is submitted to the Internet Protocol, consists of a packet header, SCTP control chunks (when necessary), followed by SCTP data chunks (when available).

> SCTP

> It also has multiplexing built right in.

It is not. You canʼt stop, for example, receiving from stream 0 during getting high-priority data from stream 9. No userlevel API allows to specify "now receive a message exactly from stream 9". Data packet TSNs are the same number sequence for all streams. You canʼt normally ignore a single data TSN for a stream payload: you have to issue SACK and brook constant retransmissions from other side. You canʼt specify receive window separately for each stream, to calm the sender down while the streamʼs receive buffer is full.

I donʼt know who and why spreads the myth it is really multiplexing, but now these stream numbers are merely another type of per-message external tag. All other is merely scam. It seems that initial design was moving toward this possibility but then something unfortunate happened.

> This means you don’t have to devise message length communication into your application layer protocol and can rely on your transport.

This is, well, tasty from it.

One channel, multiplexing, sequenced packets. It looks like you are describing SSH.

"lead to bad library design"

how?

More complex buffer management in non-garbage collected languages.

man 3 cmsg

For a glimpse into the darker corners of hell. At least you get to grip it there. In GCed languages you have to hope the library designer gave you reasonable controls for timeouts, aborting large data transfers, and releasing buffers efficiently in a bursty network environment.

See all the controls any HTTP server gives you for this. "Maximum header size", "Maximum header(s) size", "Maximum Body Size", "Request Header Timeout", "Total Request Timeout."

None of those were added by default or by tasteful library authors. They were all added to answer to specific emergent security vulnerabilities that were discovered and then were demanded by the users.

JSON's everything is "indefinite length". Also its implementations are wildly inconsistent [1] (is anyone going to read the code to figure out which is the blessed systemd parser?). Also it doesn't have integers. A lot of things with JSON will deteriorate into stringifying anyway, for example dates.

[1]: https://seriot.ch/projects/parsing_json.html

CBOR does have indefinite length arrays, maps, byte strings and text strings : https://www.rfc-editor.org/rfc/rfc8949.html#name-indefinite-...

I'm a fan of DAG-CBOR, which is a deterministic subset of CBOR with most of the weird bits excluded (like indefinite lengths, multiple floating point formats, etc.), but it can still represent any sensible JSON object.

> I genuinely dislike CBOR. Formats which require me to calculate the length of the message before sending it

CBOR is not. No full message length. Only primitive type values. Compound type values require length _in items_ (not bytes) but indefinite encoding is possible if allowed ("deterministic" constraint to message forbids it).

So technically this doesnʼt differ from e.g. JSON where you have to wait until final "}" or "]" for a long compound item, and have to install an artificial limit.

> Add in an "indefinite length" option and you've got potentially unbounded client memory usage to watch out for. > As if that wasn't enough you get extensible tags so the meaning of any message is entirely dependent on the context it was sent in.

So does any such a protocol. JSON, XML, any ASN.1, CBOR, whatever. If there are no extensible tags, there are field names - or they will be added by customer using what is allowed. Iʼve seen this in ASN.1 sequence-of pairs of name+value, directly emulating JSON-like dictionary. To limit it is not how security issues are handled.

That'd be "I can't believe it's not msgpack with extra design flaws and the name changed to my own name" CBOR?

It's not the data format that is important, but the tooling, code generation and binding quality.

Data format is absolutely important for operating system IPC. JSON is raw text. Messages occupy more RAM than a binary format, which increases your working set, and it requires at least some nontrivial parsing. This ultimately limits the number of messages you can send per second.

It's not that simple:

    [1.2,1.3,1.4]

is 13 bytes, the equivalent binary storage is 3*sizeof(double) (24 bytes) + whatever overhead exists for representing a dynamic array in your language

What's the chance the data you want to transmit is exactly 1.2 rather than 1.205338843375?

If you expect to routinely send small numbers your serialization protocol can use floats or some small fix point format.

Variable length binary number encodings are also an option.

The dynamic array overhead in a language is immaterial when discussion serialization formats.

Personal opinion: don't put floating point numbers into an API. If you really want real numbers passing them as decimal string is actually sensible. This can be done without using JSON, of course. Can also do fixed precision as well, which will store reasonably well into a VLQ.

OTOH this is pretty cherry picked. It is questionable if you really need floating point numbers if your numbers are all this small and low precision, but in actuality unless this comprises your entire message, you'll probably still lose to JSON overhead eventually anyways, whereas non-self-describing serialization like capnp has effectively no overhead by default. The advantage of encoding as doubles is that it is predictable and won't explode into taking massively more, its always the same size per number. If you want it to be smaller for some reason, compression is always an option, though I suspect for IPC it's the wrong thing to be concerned about.

> If you really want real numbers passing them as decimal string is actually sensible.

Hexadecimal, if so. Nearly any decent modern language and runtime can it.

    $ python
    >>> import math
    >>> math.pi.hex()
    '0x1.921fb54442d18p+1'

This gives exact value without any rounding effects and conversion cost.

(I donʼt count decimal floating-point as it is quite rare and its application is faint. In overwhelming most of cases, fixed point is better.)

Or an ASN.1 DER encoded sequence -- ASN.1 is more common than CBOR (being in a lot of PKI stuff) and there are already generators for DER (or JSON or XML...) for a given schema.

Which makes me wonder how performance sensitive is the task D-Bus handles?

Most task done over it seems to either fall into the "from time to time but not really performance sensitive at all" category or the "a mass of system events which can become huge" category.

> miss out on the best benefits the best benefits of using binary serialization format

I disagree (wrt. MsgPack), the biggest benefit is of binary formats is that you can ship binary blobs without any encoding (mainly space) overhead while for JSON this can easily be ~33%+ space overhead. And even for just e.g. numbers the space overhead can be bad. I'm not sure what you see as the biggest benefit but not having a minimalist structure isn't a property specific to binary formats (through more common there, and MsgPack doesn't require field names to be included). And bit fiddling micro space optimizations aren't that grate/as much a benefit as drawback. And as a system bus handles (emits) potentially a _ton_ of events saving 33% can matter I think.

A lot, D-Bus in a way has the same role as XPC, COM and Binder on other platforms, so many desktop applications use D-Bus for their plugins, now imagine the traffic of something like Thunar doing D-Bus calls for generating image previews for the new vacations album someone just created.

The obvious choice is ASN1 /s

Reply on news.ycombinator.com