Like upgrading when a data structure changes between versions. The HTTP protocol nginx serves is stateless and by comparison far simpler. Same goes for Erlang. It offers nothing more than simple function replacement, and that's not enough to handle data structure changes either.
handle_call(Request, From, State) when is_record(State, state) -> handle_call(Request, From, upgrade_state(State);
handle_call(Request, From, State) when is_record(State, state2) -> ...
Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency.
http://en.wikipedia.org/wiki/Dynamic_software_updating#Updat...
Erlang doesn't offer this check.
"Old code may still be evaluated because of processes lingering in the old code."
http://www.erlang.org/doc/reference_manual/code_loading.html...
http://www.erlang.org/doc/man/gen_server.html#Module:code_ch...
For example:
http://stackoverflow.com/questions/1840717/achieving-code-sw...
BTW, You can even support downgrade. :)
If we're talking about a gen_server, the state is per process, and once the process has switched to the new code, it won't go back, so there's no problem with old code and new state. In non gen_server code, you do need to be careful about when you hit a boundary that gets you into new code; you'd typically want it to be your process's main loop, since that usually tail recurses and doesn't leave a stack in the old code. It is difficult to reason about a situation where you call into new code, and that returns to old code; it's much better to avoid it.
The concurrent case is OK too, each process manages its own state, and upgrades it when it switches to new code. Are you thinking about changes to messages that are being passed and/or global state? In that case, like with any distributed system, you need to load in stages: first load code that can handle old and new messages, then trigger sending new messages (code load or config setting), then load code that only handles new messages.
An upgrade doesn't involve only the in-memory state per process though. It also involves state outside the process, like state on disk. Even if each process upgrades it's own state (I'm assuming the gen_server isn't limited to in-memory state; I don't know), an old process accessing from disk a data structure that differs from the one used by the new process isn't safe. You can't just upgrade old processes in stages.
An upgrade can also involve multiple processes. It's hard to upgrade all of them at once. As you mentioned, in the hardest case of all, a distributed system, loading in stages may be the only option, provided the system was explicitly designed such that old and new processes can coexist without safety issues.
Of course possible to do in-place, but I'd imagine serialization makes testing it easier. Not to mention sending error reports if something goes wrong. Having all that state in a bug report could help a bit!