Composable Network Protocols vs. Encapsulation

Interestingly, SMTP used to be an example where you could do it both ways, in relation to having a secure connection.

Originally, SMTP on port 25 did not support TLS. Later on, SMTPS came along, which uses/used port 465, and which immediately started TLS on connection open, with the standard SMTP traffic happening once TLS setup was complete. So, by choosing a different port (465 vs. 25), you (the client) were able to say "I want to do SMTP over TLS, and you [the server, operating port 465] want to do the same, so let's just do it immediately!". Nowadays, you have ESMTP on port 25, where the server advertises support for STARTTLS, the client requests startup of TLS on the session, and once the TLS connection is up, then you have your proper SMTP conversation.

See also, RFC 8314.

I was thinking about this problem while writing a SMTP server in C this weekend and came to pretty much the same conclusion. Protocols are implemented by FSMs that can be initialized and composed in various ways, but none of the protocols owns the socket. Socket is owned by the application, and it is the application that polls it and passes the events to the protocol FSMs. This way it's easy to switch to TLS on STARTTLS without any hacks, because application can decide to reconfigure the protocols stack at any time. Managing lifetime for the socket is also simpler, because app does it, and ownership is not passed to the protocol implementation.

It's work in progress, but should be easy to reason about and work with even without using coroutines.