Ask HN: Practical examples of runtime modified software | Hacker News Reader

Ask HN: Practical examples of runtime modified software

5 pointsroeles10 comments

Learning about languages like Smalltalk and Erlang, made me think about software that is modified during runtime. I've seen the Erlang video as a small demo of what's possible.

Still, I wonder how modifying a software system which runs works in practice. Therefore I'm wondering if HN has practical examples of systems that ran for a long time and we're modified during runtime. I'm curious how this works in OO languages (like Smalltalk), in functional languages and in logical languages such as prolog.

Does anyone have experience with this?

View on HN

In classical Smalltalk, there is no difference between development time and runtime. Everything, including system code modification, needs to be done in runtime. There is no difference between a running program and a development environment so writing a program in Smalltalk is a process of self-modification of the development environment to extend it to provide the functionality you want. So, for Smalltalk, it is essential to provide tools to allow that like objects identity swapping, migration of living instances to a new class definition, objects enumeration etc. The basic conceptual idea behind it is to look at Smalltalk as a living organism that is able to grow and evolve without the need for restarts/bootstrap.

You may try Pharo (https://pharo.org/features) to experience it yourself.

Thanks, I'll give Pharo a go.

While I was at WhatsApp, we used hot loading for most server updates on our Erlang cluster(s). There's tricks and traps, but if you do things right, it basically works and you can fix a lot of bugs right away without dropping connections.

The way Erlang works is any module can have two versions loaded; you can call them current and old. A fully qualified function call (Module:Function) calls into the current module, but non-fully qualified calls (like function calls within the same module, and some function references) will stay with the version they were made from, so if you're in what was current code, and a new version is loaded, current becomes old and non-fully qualified calls go to old.

You can only have those two versions though, if you load a third, any processes that are in the old version will be killed before current becomes old and new becomes current. There's ways to check if there are any processes 'lingering' in old code, and design techniques to reduce the chances of that; the easiest is making sure your loop code calls ?MODULE:loop(...) instead of loop(...), and most of the OTP looping constructs (gen_X) do that, or even if they weren't reloadable, they're doing the looping and calling into your code with fully qualified calls anyway. Tail Call Optimization is the easiest way to get the older code off the bottom of your stack, although there are others.

Of course, you can do hot code loading in lots of other languages too. In C, you can do it intentionally with dlopen/dlsym, or if your OS uses mmaped libraries, by editing libraries while they're loaded (whoops! use install instead of cp to avoid doing that by accident).

Thanks for providing some insight in how WhatsApp did this with Erlang!

Is all of the logic to support runtime upgrading inside Erlang/OTP, or does the software you write have to be written (partly) in mind with runtime upgrades as well?

Also, thanks for the hint regarding cp/install. I figured that the kernel would recognize an open descriptor to the old file and keep that old copy until all references are gone.

OTP has some structures to help with upgrading that we didn't use. Mostly, you have to keep upgrading in mind when you make changes. Easy ones are things like if you add another field into a record (like a struct) that you're using for process state, you need to still handle the old version of the record, since it will be loaded into processes with old state. This is similar to what you need to do with APIs (local library or remote calls to other computers), but it's a little unusual to consider it in a running process.

For cp/install; cp will generally write over the existing file, but install unlinks and then writes a new file. If you've mmaped the file, you get changes in your process from cp (copy on write doesn't apply to mmap), but not with install (the old file is kept around until it's no longer referenced).

Perhaps you could say how you think it works in practice.

I don't know! Which is exactly why I ask the question.

I think it would be a nice option to have, especially on systems that are supposed to run for a long time.

I _guess_ it often involves bringing parts of the system down, or living with multiple versions of code inside your system.

I know a little bit about how Erlang does it, but I have never seen a real-world example using Smalltalk. I suspect there must be, since Alan Kay emphasizes "growing" software in some of his talks.

Helps to have some idea of what you might or might not have assumed.

As you say "bringing parts of the system down, or living with multiple versions of code inside your system" — at least momentarily some other modifications to data will be paused. In practice, the data is probably in a relational database and that will determine how the transition is done.

Here's an experience paper from long ago: https://news.ycombinator.com/item?id=31162816

----

When everything is within Smalltalk — create-update-delete methods to change behavior; to change state, find all instances, copy data to new objects, magically become: all references to reference the new objects instead of the old objects, check that GC removed the old objects, and save.

These Computer History Museum videos might show something like that: https://computerhistory.org/blog/introducing-the-smalltalk-z...

Thanks for the exhaustive response!

Not smalltalk, but my KaithemAutomation software is pure Python and heavily built around runtime modification(https://github.com/EternityForest/KaithemAutomation), and is designed to run for months, although it's not meant to be modified literally while running production(Not that that stops us from making last minute fixes...).

The big problem with it in Python is state. If you have a reference to something, and that something changes, and you have a reference to the old one, that should be a singleton, you are probably in an unhappy position.

The only way it can work in real life(In Python at least) is if you carefully design for it, and don't pass around objects and callbacks and stuff.

I have gone through several iterations of abstractions trying to make updatable objects and such to hide this from you, and none were very good.

I eventually settled on a message bus, and a data structure I'm calling a Tag Point adapted from the SCADA industry, which is like a variable, but you can subscribe to it, and spy on the value from the web UI, it stays around as long as there is a reference to it, and it's guaranteed that any tags with the same name are the same object, and a bunch of other stuff.

Files of code are essentially meant to be used like stateless microservices if you expect to update one, and if you want to access shared stuff, you make sure to not hold a reference to it.

A lot was(and some still is) based on weak references, and those can trouble and should probably not be relied on for correctness if possible.

They aren't the worst things though. I have code to run a few GC sweeps when a file is deleted, and it is extremely rare that anything stays around when it shouldn't.

Good enough for development, good enough for emergency fixes, not the best for regular updates to running systems, although it's 99.9% fine, and I can't say I really worry about it(But future versions should be more deterministic and more suited for live edits as a regular practice).

Erlang seems to like its functional-ness and I imagine that's a huge asset. Language level support is definitely a good thing.

Random unstructured code doesn't seem to work well with live updates.

If you are working in a general language like Python you really want to have your engine always know exactly what's going on, what subscriptions come from what module and which function is replacing what, etc.

You want to deterministically always be able to list any changes that a module made to anything else(Like subscribing to a function), and undo them.

But... weak refs work well enough.

The classic solution to tech problems is to reboot, live updates are kind of the opposite of that. Your new code has to perfectly pick up with what you keep from the old state, and if the old state is invalid somehow you have to deal with that too.

I've never heard of smalltalk as a live update language(In the Erlang telephone exchange sense), just that you can do interactive development in it, which is a lot easier.

I think if I intended to seriously to true live updates(Like phone exchanges not dropping active calls), I would really appreciate tools made for that.

Kaithem also does have a module with a visual script explicitly meant for changes in production. It's very limited and opinionated, essentially a state machine with event triggered actions attached to states, and variables at the state machine level.

That kind of transition rule system for simple tasks is very easy to live update. State is well separated from rules, and rules are simple and easy to parse programmatically, to do things like clean up after yourself if an event listener needs resources.

But just about any interpreted language works for interactive development.

A lot of people like FORTH for that, which I don't have any interest in learning but some love it.

Reply on news.ycombinator.com