Is this really true? I remember seeing an article[1] recently on using an iptables hack to prevent dropping connections when reloading haproxy. Does nginx actually provide zero-downtime configuration reloads?
[1] https://medium.com/@Drew_Stokes/actual-zero-downtime-with-ha...
Remember, fork()ed and exec()ed processes inherit file descriptors (except those marked CLOEXEC), including the listen() fd. Pending connections will queue in the kernel until userspace calls accept() on the listening fd.
So one simple model is to stop calling accept(), cleanly/quickly finish up current connections, set an environment variable to tell the future instance that the listening fd X is already open, and exec your own binary again.
A more complicated one is to fork, have the (identical) child just finish the current connections, while the parent execs itself similar to above. (The client connection fds should be marked CLOEXEC in this case.)
With a more complicated service with more moving parts, libraries, threads, getting the above to work out is more complicated. But that's basically how you want to do it.
systemd already supports creating listening sockets, but its mode of operation is more similar to inetd. I think supporting something like what my proprietary launcher program does would require very little changes.
Another advantage of passing the listening socket like this is that the server process can be in a private network namespace, without requiring any type of NAT setup or port mapping.
Like upgrading when a data structure changes between versions. The HTTP protocol nginx serves is stateless and by comparison far simpler. Same goes for Erlang. It offers nothing more than simple function replacement, and that's not enough to handle data structure changes either.
Of course possible to do in-place, but I'd imagine serialization makes testing it easier. Not to mention sending error reports if something goes wrong. Having all that state in a bug report could help a bit!
handle_call(Request, From, State) when is_record(State, state) -> handle_call(Request, From, upgrade_state(State);
handle_call(Request, From, State) when is_record(State, state2) -> ...
Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency.
http://en.wikipedia.org/wiki/Dynamic_software_updating#Updat...
Erlang doesn't offer this check.
"Old code may still be evaluated because of processes lingering in the old code."
http://www.erlang.org/doc/reference_manual/code_loading.html...
http://www.erlang.org/doc/man/gen_server.html#Module:code_ch...
For example:
http://stackoverflow.com/questions/1840717/achieving-code-sw...
BTW, You can even support downgrade. :)
If we're talking about a gen_server, the state is per process, and once the process has switched to the new code, it won't go back, so there's no problem with old code and new state. In non gen_server code, you do need to be careful about when you hit a boundary that gets you into new code; you'd typically want it to be your process's main loop, since that usually tail recurses and doesn't leave a stack in the old code. It is difficult to reason about a situation where you call into new code, and that returns to old code; it's much better to avoid it.
The concurrent case is OK too, each process manages its own state, and upgrades it when it switches to new code. Are you thinking about changes to messages that are being passed and/or global state? In that case, like with any distributed system, you need to load in stages: first load code that can handle old and new messages, then trigger sending new messages (code load or config setting), then load code that only handles new messages.
An upgrade doesn't involve only the in-memory state per process though. It also involves state outside the process, like state on disk. Even if each process upgrades it's own state (I'm assuming the gen_server isn't limited to in-memory state; I don't know), an old process accessing from disk a data structure that differs from the one used by the new process isn't safe. You can't just upgrade old processes in stages.
An upgrade can also involve multiple processes. It's hard to upgrade all of them at once. As you mentioned, in the hardest case of all, a distributed system, loading in stages may be the only option, provided the system was explicitly designed such that old and new processes can coexist without safety issues.
its not that hard to do if the program is made for this, even in C, its pretty hard to add when it isnt tho since you need a compat. layer.
Sometimes the new program simply has import functions for the old memory layout.
The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code.
That would imply that two separate processes are bound to the same port right? I thought that was not possible.
Erlang can have two versions of a module present in memory at a time. At the moment of a code upgrade, the processes working with existing connections continue to use the old code while newly accepted connections are passed to the version of the module that was loaded in between connection accepts.
(e.g. http://stackoverflow.com/questions/14351147/perl-passing-an-...)
One way of doing such upgrades is then for the existing code to fork() then exec() the new version. The existing old code closes the original server socket, which leaves accepting new requests to the new code. When all in-flight requests served by the old code is finished, that process can exit.