Linux implementation of Homa, a protocol to replace TCP for low-latency RPC
micahlerner.com
micahlerner.com
I rolled my own framework for fun that did REST-style communication but sans TCP, HTTP and JSON.
This improved latency and efficiency by couple orders of magnitude. I would like something that did this that I could use directly without having to write my own framework.
- was it over a LAN or WAN?
- how were the latency improvements measured?
- was there a UI on the front end or was this strictly a backend to backend scenario
I measure latency on a client, on service and on using some very expensive, cut-through switches. This shotgun method lets me lazily detect where the latency comes from with a single look. I understand not everybody has access to cut-through switching, but for most uses port mirroring is available on a lot of hardware
Strictly backend.
* https://en.wikipedia.org/wiki/Open_Network_Computing_Remote_...
QNX native networking over IP (protocol 106) is probably the closest to Homa. It's an RPC system, intended primarily for local networks, although it will extend across the Internet. Message based. No head of line blocking between different RPC calls. It's still used in Blackberry/QNX.
Is there a spec for Homa? Did they get a new protocol number from IANA? [1] This needs an RFC before it gets out into the wild.
[1] https://www.iana.org/assignments/protocol-numbers/protocol-n...
That said QUIC is rather flexible since implementations can be done in userspace, and a lot of what this protocol is about can probably be done by running a custom QUIC library. The GRANT messages can be emulated with window size increments on streams, and the retransmissions just look like different ways of handling ACKs - so this might be emulateble by having some custom ACK behavior and an infinite congestion window on the sender (bandwidth is limited by the receiver using window size increments). The usage of hardware-supported priorities might be a bit more tricky to map into QUIC - but it might be doable too if stream data frames can be sent in different UDP datagrams using different priorities.
Whether that really works better than the default setup? I'm not convinced. The nice thing about standard TCP and QUIC is that it works with all kinds of traffic. Congestion controllers will allow coexistence with other traffic on the links ("oh, there was a 5Gbit/s log upload or software update download which used 50% of NIC capacity?"), and the congestion window adjusting itself over time will also mean that "small RPCs can be sent without waiting for any confirmation". It even means "medium RPCs can be sent at once, once we learned about the link capacity (congestion window)".
I know that SCTP isn't very popular, but it is an old standard and solves the ahead-of-line blocking issue. Linux does support SCTP, and maybe more expensive routers support it as well (honestly, I haven't experimented much with SCTP on networks...).
SCTP is built on top of IP. So the main issue with SCTP is firewall / NAT / etc. etc. which are built on top of TCP / UDP instead. Homa, being an alternative protocol, would be very similar to SCTP (in that routers wouldn't work on it), except its decades younger.
UDP is just a port number + rarely used fragmentation layer tacked on top of IP.
So any protocol designed on top of IP can almost certainly be used on top of UDP instead.
Comparitively, QUIC layers encryption directly into the protocol. I can't say how it deals with the compute cost because I'm not familiar with the protocol at that level, but it's a clear design decision - since handshakes are expensive, including encryption in session establishment has significant benefits. Perhaps encryption configuration is expected to be handled in a side channel, but it's odd that in a discussion so centered on what the next layer up is doing it's completely forgotten this critical aspect.
Layers of communication are a thing for a reason, it allows each piece of the communication to upgrade independently. Instead of writing encryption into the spec, its more useful to write encryption on top of the spec as a 2nd layer.
TCP is very old. You would not do that today, and indeed they didn't, QUIC's cryptography is built in.
Two reasons to want cryptography built-in. One in some sense political, "Pervasive Monitoring is an Attack" is BCP #188. We must defeat such attacks on the Network by encrypting everything possible. The other is pragmatic, middleboxes induce Protocol Ossification, making innovation difficult or impossible, by encrypting everything we prevent a key source of ossification, if the middleboxes can't understand the protocol they also can't rust shut extension points.
Similarly, you could make an argument that because this interconnect tolerance is so low, pervasive monitoring has a different risk profile. Two servers communicating via TOR thru a 40G NIC using Homa don't even have an opportunity to pass the packets elsewhere in the hierarchy to be snooped by someone else, unless it's just streaming packets directly into the three-letter-agency van in the parking lot I guess. For all practical purposes in a system like this the requirements are so tight you can only afford direct links between systems (or something close to that) if you want to maintain the desired operational behavior, so third parties appearing between two points might not be possible, much less likely.
That said I think including encryption would be good. But the reasons would be very pedestrian IMO: for one, it ticks off the annoying compliance checkboxes (both political and social) and means there's a single design for anyone who implements Homa to follow, so people don't have to painstakingly re-create it. And a corollary of that checkbox tick is that it nips threads like this one in the bud regardless of all the above. :P
Something like FASHIONCLEFT. Your smart managed switch's firmware squirrels away summaries (e.g. it sees 400GB of data about Project Smith and it notes [400GB, Project Smith]) and then later squirts such summaries over legitimate links to distant nodes, but passing through another device with compromised firmware, and the other compromised device removes the extra data and gives it to the NSA.
The NSA values this because plausible deniability is essential to their work, if you realise you were compromised you are likely to blame the destination not them.