Comment by Veserv | Hacker News Reader

Comment by Veserv | Hacker News Reader

Parent Full thread

1. I do not see where they discuss that in the RFC. While that would work, it sounds awful.

That is a sender-driven resend and as such relies on the sender identifying the “send lost” condition. However, the entire protocol is designed around not doing that and thus you can only feasibly rely on “should have received a reply by now” as your timeout.

But Homa is intended to be a RPC protocol. So you send to server, wait for server to process command, then wait for server to send reply. Your timeout depends on the variable and heterogeneous command processing time.

Even if you were able to give a separate correctly tuned timeout for every possible RPC that is still awful. Any RPC with long processing time should not trigger the timeout until the expected reply time, but if that is far larger than the RTT then you are waiting a tremendous amount of time.

For instance, a select query that is only a few bytes long (and thus fits in one packet) could take seconds on a large database even though the database server is physically nearby and only microseconds away. In that case you would have to wait seconds before timing out instead of just microseconds like a ack-based design could achieve. The worst-case transport latency becomes application-specific instead of related to the application-independent transport parameters like RTT. That is troubling.

2. TCP can be forgiven for that given it predates asymmetric cryptography and even DES. Not considering it relevant in a new protocol this side of the millennium is much less reasonable.

4. MsQuic at 7.5 Gbit/s: https://microsoft.github.io/msquic/

You keep talking about Gbits but Homa is about latency.

There is nothing in the design that supports low latency relative to a modern protocol like QUIC. Basically the entire “latency” advantage it has over TCP is just that it does not head-of-line block which QUIC also does not do.

And then in basically every other respect it is worse including compatibility.

You keep changing the goal posts, typical

I'm not sure what you're getting at with your point 1. Any implementation of RPCs, over any protocol, faces the same timeout issues. If it takes a long time to service an RPC, the client will naturally wonder if something broke. Either the client has to send some sort of "ping" to make sure the server got the request and is still alive, or the server has to send "keep alive" packets to let the client know it's still alive. If most RPCs finish quickly then the ping interval should be short in order to detect packet drops quickly; but then a large number of pings will be required for RPCs that take a long time to finish.

I don't see anything particularly Homa-specific in these issues. A protocol like TCP that doesn't implement RPCs doesn't have to face these issues; it punts them up to the application, so the application has to deal with them. Homa implements all of this in the transport, so applications don't have to worry about it.

Example: 1 ms RTT with 1000 ms command processing time.

In a lossless RPC the client should expect the reply in 1002 ms. When should the client resend the request if it got lost?

Given that the client would not expect to see the first byte of the reply until 1002 ms and receives no other feedback until then, they should not issue a resend of their request until 1002 ms.

That means if the network lost their first request, they will not send a resend until 1002 ms and then get the reply to their resend at 2004 ms.

In contrast, in a ack based approach, the client would expect a server ack in ~2 ms as the server gives feedback on receipt rather than on processing completion/reply start.

That means if the network lost their first request they will not send a resend until 2 ms and then get the reply to their resend at 1004 ms, nearly half the latency.

In ack based approaches, packet loss requires one additional RTT latency to recover per repeated packet loss. But Homa incurs one additional RTT + processing latency to recover per repeated packet loss.

As a secondary downside, this requires the client to hold Tx resources for RTT + processing time since the client does not know when the server has received all of the data. In ack-based approaches, you learn the server has received all of the request when it acks all of the data in the request which is on the order of the RTT and thus can release Tx resources at that time. This is similar to the reason Homa has ACK/NEED_ACK packets except applied to client side resources instead of just server side resources.

The case you are describing is more like what if the server explodes, but you still need the results/modifications related to your RPC to occur. In that case, whether the Tx got through or not is irrelevant because your RPC still did not complete and you need it to complete, so there is no advantage to the ack-based approach. But by that logic, what if it explodes before persisting changes? Or what if all your servers explode? At some point we should call that out of scope for a transport protocol.

I prefer setting that line at “bytes got from one end to the other”. If you want higher guarantees then you can layer that. As you point out, it is easy to just layer a “server dead” timeout ping which basically solves the problem in the same way Homa needs to.

I think you may misunderstand Homa's behavior. In general there's no reliable way to predict how long it will take to service an RPC, so Homa doesn't try. Homa does quick resends using the same timer that's used to detect packet losses (i.e. a few ms), so the results will be pretty much the same as what you described for an ack-based scheme. Of course, if an RPC takes more than a few ms to complete then there will be an unnecessary resend request, but I don't expect this to happen very often. And, it won't have a significant performance impact because the RPC is already consuming many ms of CPU time for its execution; a few microseconds to handle a resend is insignificant.

I agree that a transport cannot hide all possible failures and that at some point things have to get kicked back to the application. But if the only guarantee provided by the transport is byte delivery in one direction, that makes things significantly harder for apps: every app now has to implement its own timers and pings to detect server failures. This basically duplicates timers and retries already present in the kernel. I think Homa's approach makes life easier for apps. There's a clear and simple division of responsibility. Homa detects all failures, including both lost packets and server failures. It recovers from packet loss but not server failures. Server failures are reported to the app, and the app decides how to handle them.

And, Homa's approach cuts the number of packets in half in the common case of short RPCs with fast service times. With acks it takes 4 packets for each RPC: one for the request packet, one for its ACK, one for the response packet, and one for its ACK. With Homa there are only 2 packets in the common case: one for the request and one for the response. The response implicitly acks the request, and the next request acks the previous response.

Hmmm, I see now that QUIC can delay ACKs and piggyback them on subsequent data packets, so it looks like it can get similar efficiencies to Homa.

I agree that Homa holds Tx resources for longer than would be needed in an ack-based scheme (I don't see that as particularly problematic).

Reply on news.ycombinator.com