How Copyover MUD Servers Worked
jackkelly.name
jackkelly.name
It felt absolutely magical to be able to hot deploy changes without kicking everyone off of the server - but felt even more magical to have gotten it to work.
The MUD community was a fun early introduction to open source. People (many of them probably "kids" like I was at the time) sharing various patches and features. It felt so cool to release something and have other people use it and provide feedback. Like the author says - at some point the MUD itself became a lot less interesting than the programming.
They still are these days. There's plenty around, and there's even more MUD-likes (GUI) that are open-source and played/developed by hundreds & thousands.
That version just used exec, and closed all files but network descriptors already logged in, the mapping of fds -> login names was saved in a file. When the new copy started up, it would log the users on existing file descriptors. Today, using explicit file descriptor passing (so you don't accidentally keep files open) or a long-running proxy would be preferable.
Back then C/C++ were often used by the developers, and we were at best CS students. There were surprisingly few segmentation faults, but I remember a few mysterious memory corruptions...
according to the patches in the source posted on your website the main work was done 1996-1997 ;-)
i am actually surprised that this didn't happen earlier, given that Diku itself was inspired by LPMuds which could do live code update already in the early 90s. of course the motivation for Diku was to produce something more stable than LPMud, so maybe they didn't think that live updates were a good idea in the first place. (that said, i don't remember LPMuds being unstable myself, but i only played from about 1992 at which time it may have improved)
It originated (more than 20 years ago) as a fork of SWReality, which is itself a fork of SMAUG, which comes from DIKU, and so on and so on.
We (myself and a lot of other awesome volunteer coders over the years) have made some pretty major modifications to the codebase. It all runs on PostgreSQL now (instead of flat text files with a custom parser), it has a built-in Lua scripting interpreter for extensions and in-game interactions, it has deep integrations to our Discord community, it has sidecar web services, and much much more...
By the way, you have broken link on the `Learn More About the Economy` link in the home page. Should just have it redirect to `/crafting-the-galaxy`
Kind of funny how many people from LotJMUD or SWC I've met in the real world who work in tech now
These days you could probably spawn your child process, test-boot it and then mmap bits of it back into your process and then unmap your old code pages once you're certain it boots. Or given how cheap cycles are, test-boot it and then throw that away and spawn the new executable if you're happy with the results.
even better, LPC has been rewritten into pike, a general purpose programming language that retains the same capability. in the roxen webserver i can write and reload modules at runtime. this works efficiently because the lifetime of any object instance is limited to each http request. when a module is reloaded http requests already in progress are not affected, only new ones.
in the object storage server open-sTeam also written in pike a more advanced method was developed using proxy objects that can update object references and thus allow the updating of code without breaking the references. i am still using that to host my own websites.
to this day i have yet to find any other language with this power. smalltalk can do it and i believe lisp too, but that's it.
The last maintained LPMud fork that I know of is LDMud https://github.com/ldmud/ldmud . Would be cool if there are others.
The BEAM vm has a lot of the functionality you mention from LPC and Pike. Erlang being from the 80s—predating LPC—and also from Sweden, I imagine that Lars Pensjö and later Fredrik Hübinette et al may have taken some inspiration there.
which is quite ironic given that CircleMUD is based on DikuMUD which was created to be more stable than LPMUD.
Erlang is one of the languages i am still very curious about. hopefully some day i'll get an opportunity to work with it
I hear that class/skill variety was often greater on LPMuds because it was so much easier to code up custom behaviour. Do you know of any especially good articles on LPMuds class (in the OO sense)/object hierarchies and the software architecture of mudlibs?
On boot as it's loading players, their file descriptor ID was saved, so it loads it up and resumes talking to the socket. It also loads the ID of the server socket descriptor. No need to send information to a new process as it just loads the previous game state.
If you're curious: https://github.com/borlak/acmud/blob/7c2442dfccfc28364fc399a...
And: https://github.com/borlak/acmud/blob/7c2442dfccfc28364fc399a...
Kind of drifted away for a couple decades during college and after, as other things filed up the time.
I came back decades later, after going to a memorial service for a friend who died untimely of a serious medical condition, and seeing that a bunch of the people there were from her online community. They talked about the MUSH she hung out on had been a real lifeline when she was bedbound for immune-system reasons -- it was really, really cool to see, and I went back and checked out my own place and met up with folks again.
During 2020 a TON of people all had the same idea and all logged in to my place again. There was a brief resurgence of activity (from dozens of people online to a hundred+). Very few new players, but very cool to see people who were all, more or less, the same cohort -- just grown up now. Folks have slowly drifted away again in the past couple years, and that's fine too. I'm glad it's there.
It's nice to have these subcritical, human-scale online communities. Not everything has to be a subreddit or even a 10,000+ person discord - you can just hang out on a server!
https://www.linnaean.org/~lpb/muddex/mudline.html
See also: Burka's MUDDEX (1993), curating listings of the erstwhile MUD servers as NSFnet ruled the USA
https://www.linnaean.org/~lpb/muddex/index.html
Burka was known to her fellow players as "ashne" - stylized in lowercase - on Tinymud Classic & Islandia, hosted by CMU prof Jim "Fuzzy" Aspnes, among other servers
https://en.wikipedia.org/wiki/James_Aspnes
several interesting hacks we collaborated on:
A TCP port concentrator, implemented by "leet". This add-on front end multiplexed connections with separate processes, allowing us to overcome the 64 file descriptor hard limit, per process, on Unix. Nearly 256 players could participate!
Na Choon Piaw, while working for Bell Labs Research on the East Coast, added a programming language that resembles Forth, releasing TinyMUCK 1.x and TritonMUCK test bed server. Piaw worked with a VAX server, 7800 I believe.
https://en.wikipedia.org/wiki/TinyMUCK?wprov=sfla1
also, Jon "Stinglai" Blow from Berkeley designed a string interning method for TinyMUCK 2.x that saved plenty of memory, by deduplicating ASCII strings in memory.
Me? I assumed many interesting names over the years. Primarily "ChupChup" or his plural counterparts, "chupchups", or also "Lucretia", the statuesque goth woman wearing "boots of the deepest black". But in Real Life, (RL), I remain Robert Earl.
Hopefully they gained some appreciation for how much work goes into these.
Sigh.
A big challenge lies in designing a system where every user has creation and modification rights, without breaking the system. Systems based on the https://en.wikipedia.org/wiki/Object-capability_model for access control and those that can monitor and set computational resource limits internally can prove very useful there
One of the weirdest features of the language in retrospect was that iirc you could use object reference literals in your code, since every object instance had an id in the database (eg #1234). This eventually necessitated a special recycler, implemented inside the MOO, to make sure that IDs got reused when objects were deleted.
I'm not positive on the history of the recycler, but I think you have it backwards. It actually exacerbated the problem of using literals in code because you couldn't guarantee that the object you wrote in the code is the same object 20 years later. This created security concerns (imagine you hard-code an object number into your code and now the object is owned by a completely different person who can write their own code on that object. So your code called #123:innocent_function(), which the new object owner has programmed to do something malicious) at worst and broken code at best.
I assume the reasoning was two-fold: First, huge numbers are a pain to type. And in MOO you do end up typing object references a lot. And second, in the early 90's, memory was at a premium. Even a recycled object is using some memory (it still has a space in the object list).
Anyway, if anybody is interested, MOO is still kicking! There's a fork called ToastStunt (https://github.com/lisdude/toaststunt) that offers some slightly more modern conveniences. And the community has, mostly, converged into the ToastStunt Discord channel.
But in my experience (which may vary from your own), this takes significant investment and you’d need a very strong incentive or it would need to be a passion project of some sort.
Get a signal, do the thing (fork, then exec, then copy state, then you're done).
But, there's other options. If you want to keep long running connections, but there's not really shared state, you can just fork and then move the socket to the new server and let the old one drain out and exit when it's done/ after you waited long enough. Using a BEAM language like Erlang or Elixir gives you hotloading as a fundamental tool, and otp has some suggestions for how you could use that. You can do hotloading in C with dlopen and dlsym/dlfunc, but you've got to structure your code carefully to make it work, and you may be stuck with a fixed outer loop, depending on how creative you are at writing weird C. Java has hotloading too, at least in some vms. I've built hotloading in Perl, but I don't think I'd be proud of my code if I looked at it again.
https://github.com/michaelprograms/nightmare-residuum is a fine starting point, and I'd recommend playing discworld.atuin.net 4242 to get some ideas.
(you could probably do the same thing by persisting state into a file, though)
Also, I looked it up, and apparently on Linux it's a 65536 byte buffer.
It's stuck me recently how much I miss that kind of programming. Everything I do now is hidden behind so many layers of abstractions, toolkits, libraries and environments that it all feels a bit divorced from what's actually going on. I need to come up with an excuse to do a bit of low-level coding again.
Other options (some of which the article briefly alludes to) include POSIX shm_open, Linux memfd_create, Linux O_TMPFILE. So long as you get either an FD or a filesystem path, which can then be passed over the exec in either argv or envp
If you use a custom memory allocator with its own heap, you could then store that heap in a shared memory segment and then remap it after the exec. I guess the risk is the memory layout may have changed so you can’t remap it at the same address, in which case you either crash or have some code which rewrites all the pointers in the heap (potentially very painful to do reliably…) Or you could just not allow raw pointers in the objects in that custom heap, requiring them all to be offsets from the heap start
You can keep the process alive if copyover fails by writing to a temporary file and designing the server so that you get a "rescue console" if it fails to come up. Then you could inspect the dumped file, see what's going wrong, and exec() into another binary to try again. You could even use sqlite as your state file, which would let you interactively explore and edit it during debugging.
A custom allocator sounds like an extremely interesting approach. I'm still not sure whether I'd want to rely on struct layouts not changing, but if you use it diligently you at least make it easier to find everything you've allocated.
Less-portable or less-reliable alternatives include `memfd`, a deleted-but-still-open tempfile, etc.
To avoid problems with failure after `exec`, it is very important to fork+exec the server first just to see if it actually starts. If you don't need the PID to remain the same you can actually just use that instance without the other fork or another exec (this still keeps the same PGID). Note that "failure after `exec`" can be fairly sharply divided into "error in C runtime (usually, library compatibility)" and "error in something `main` calls", which require very different approaches.
---
Aside: I've had catastrophic failures using the "save after fork" approach (mentioned later in the article) when the child process crashed before the save completed, and the parent just kept trying to have the child save (I also managed to completely crash GDB while investigating this). It is very important for the parent to commit suicide if the child can't do its job, to minimize the amount of lost data.
The other thing I would recommend to anyone working with this kind of thing: use systemd and commit all the way; it will make your life 1000% easier than trying to reimplement it badly yourself.
Windows is a notable platform not supporting socketpair.
Some languages offer socketpair on Windows, e.g. Python, but they are implementing it themselves using TCP loopback connections.
Somewhat irrelevant to this discussion given Windows lacks exec (starting a new executable requires a new process which gets a new PID), and in practice lacks fork too (the low-level NT API has undocumented support for forking, but most of the higher-level APIs get confused by it and break when you use it, making it unusable by the vast majority of applications)
Well, you can do exec() on Windows if you implement it yourself in user space, see e.g. https://github.com/polycone/pe-loader (a bit of a dated project, 32-bit only, but the I believe the same principles apply to 64-bit)
Cygwin uses a hack – exec() creates a new Windows process, but every Cygwin process has two PIDs, a Windows PID and a Cygwin PID, so exec() changes the Windows PID but keeps the Cygwin PID the same – so it looks like an exec to Cygwin processes, but just like an ordinary spawn to non-Cygwin processes
Less insane approach: if you put most of your code in a DLL, you can unload the DLL and then load a new version of it
https://github.com/avidal/nakedmud/blob/fcab059439515be9ad93...
For a long time you could reach me at stephan@mud.de :-)
I bet that at this point, a typical Kubernetes microservice deployment ends up emulating a subset of Unix, with UUIDs in place of FDs and PIDs.