FWIW, ASN.1 BER supports an indefinite length encoding (though DER does not; ASN.1 JSON and ASN.1 XML do as well)
I was thinking about this more and I think FTP actually had a lot of the right ideas here. You want two channels. One for fast interactive command messaging and a second one coordinate especially just for bulk transfers arranged over the command channel. The design flaw in FTP of putting these on two separate ports put it immediately in conflict with firewall best practices so I think it went mostly unnoticed that it fundamentally is a good arrangement.
What you want is one channel, sequenced packets, with a defined maximum message length that is negotiated at startup and never changes during the life of the channel. This should probably never be more than 65k. There should be a known length packet type, and an unknown length packet type, with any attempt to send more than the negotiated maximum triggering an error and disconnect.
If you do need to send more than the negotiated amount you should open a new connection, in a bulk transfer mode, that after the initial handshake, has no protocol and is merely an associated stream of bytes that are wholly uninterpreted by the middleware layer other than to help you associate it with the sequenced packet that requested its initiation.
You'd actually be able to use TCP (with DSCP even), mostly avoid head of line blocking and multiplexer latencies, and have a reasonable security guarantee in the sequenced packet mode, and never have a protocol which pretends that 4GB strings in the middle of a packet are necessary or even a good idea to "support."
The downfall of this is that it would be much harder to implement it on a serverless architecture and would be nearly as complicated as a protocol as WebSocket ends up being. It might be worth playing with as a concept anyways.
For a localhost/UNIX domain protocol, it might work since most of these considerations are significantly reduced.
As to DSCP, it might be useful across controlled networks but in general, in my experience it's not really ever honored broadly.
Back on the original topic, when I was building this for my Linux distribution I ended up just using basic JSON with TLS (for client certificate authentication) though since having authentication for remote management was a goal, and once I was already having to perform a TLS negotiation then the PDU consideration for performance wasn't something to really spend too much time on.
You would need to tightly control and multiplex the messages yourself, needing to do this in one connection or using something like TCP priority flag.
Personally I just think that TCP is a shitty protocol for building applications. In almost every use case either UDP or SCTP are a better choice. With raw UDP datagrams you aren’t guaranteed delivery but it’s great for the kind of telemetry where the last message in is what matters. You can also build quite flexible stuff on top of it.
SCTP gives you in sequence reliable datagrams and congestion control. This means you don’t have to devise message length communication into your application layer protocol and can rely on your transport. It also has multiplexing built right in. If it had a built in checksum it would truly be ideal. Not sure if something like secure communication really belongs at this level but if it had that I double we would ever use anything else. From SCTP’s Wikipedia page:
> SCTP applications submit data for transmission in messages (groups of bytes) to the SCTP transport layer. SCTP places messages and control information into separate chunks (data chunks and control chunks), each identified by a chunk header. The protocol can fragment a message into multiple data chunks, but each data chunk contains data from only one user message. SCTP bundles the chunks into SCTP packets. The SCTP packet, which is submitted to the Internet Protocol, consists of a packet header, SCTP control chunks (when necessary), followed by SCTP data chunks (when available).
> It also has multiplexing built right in.
It is not. You canʼt stop, for example, receiving from stream 0 during getting high-priority data from stream 9. No userlevel API allows to specify "now receive a message exactly from stream 9". Data packet TSNs are the same number sequence for all streams. You canʼt normally ignore a single data TSN for a stream payload: you have to issue SACK and brook constant retransmissions from other side. You canʼt specify receive window separately for each stream, to calm the sender down while the streamʼs receive buffer is full.
I donʼt know who and why spreads the myth it is really multiplexing, but now these stream numbers are merely another type of per-message external tag. All other is merely scam. It seems that initial design was moving toward this possibility but then something unfortunate happened.
> This means you don’t have to devise message length communication into your application layer protocol and can rely on your transport.
This is, well, tasty from it.