Hacker News Reader

Comment by rwmj | Hacker News Reader

Intrigued by how this can work in the face of changing data structures. eg. You start by storing some data in a linked list. However later you realise that's too slow so you switch to a hash table. There seems no sensible way to load a new object file and have it access the old data without some kind of in-place upgrade system (and what if two modules want the data in old & new formats?)

Edit: He does mention some kind of data persistence system, but I think he vastly underestimates how complex and insidious this problem will be.

View on HN

Erlang does it, but it requires an extensive system, and the application programmer must support it, too: [0], [1].

[0] https://www.erlang.org/doc/design_principles/release_handlin...

[1] https://www.erlang.org/doc/man/gen_server.html#Module:code_c...

It's... cumbersome, let's put it this way, and mind you, Erlang makes it happen at "safe" points, when the code being replaced is idle.

> some kind of in-place upgrade system

For Common Lisp, here's a toy example[1], and spec[2].

[1] https://malisper.me/debugging-lisp-part-3-redefining-classes... [2] http://www.lispworks.com/documentation/lw70/CLHS/Body/04_cf....

> changing data structures

liballocs is an approach for providing high-level facilities such as changing structure layout to a a low-level Unix environment.

https://github.com/stephenrkell/liballocs

Nobody intended to give an answer to an unsolvable problem, but to solve practical problems. If you redesign data structures, most likely you'll have to restart the program. So what?

I agree, this is still very useful in something like game development where you want to maintain state and you’re well beyond the point in development where you’re changing data types!

That's simply unacceptable for some systems, like if you're actively routing phone calls.

It's acceptable (and required) for others. Which is my point.

If a restart is unacceptable then live-patching a data structure change at runtime requires a #YOLO approach to developement and ice in your veins

Well, I can't say too much, but it's not as bad as you might think. It's extremely annoying, but doable.

It's really not that different from doing a DB table update.

Depends on the platform. Erlang was specifically designed around telecom systems that needed maximal uptime, and the language and runtime collaborate to support hot patching with a distinctly non-YOLO approach.

Interesting, Erlang comes up here on HN rather frequently but I haven't really looked into it. I think I should though, any resources you can specifically recommend?

Funny you should ask…

I haven’t checked these links in quite some time: https://gist.github.com/macintux/6349828

And if you don’t mind videos, here’s my talk from Midwest.io (RIP) on the philosophy behind Erlang & its VM: https://youtu.be/E18shi1qIHU

Live patching is cool but it's something of a mythical beast. I hear lots of people talking about the wonders of live patching but very few people reporting on actually using it in their own production systems. If you're building an ultra-reliable system, then you have to somehow make it robust in the case where (some of) the hardware executing your code fails. Once you've covered that case, there's not much need for live patching.

I think it depends on which level of live patching you're talking about.

At Basho, we would load module fixes into Riak instances for customers...I hesitate to say "fairly regularly" because it's been long enough and I had limited visibility into it, but it certainly seemed like a routine operation.

The full "upgrade the entire application" hot loading definitely requires more planning than most companies will ever be willing to do.

http://www.lispworks.com/documentation/HyperSpec/Body/f_upda...

That's how you'd handle it in Lisp. Smalltalk has something similar, but I'm less familiar with the internals for it. No restart needed, but it requires a runtime that is able to track data instances by type. You could build that onto C, though. Probably with a custom allocator that was passed some symbol indicating the structure type so it could report on instances for executing an update. Would be hard, though, to do this without invalidating pointers when the size changes. You'd probably end up with an extra level of indirection as a consequence to make it simpler (for the runtime author, not for the end user).

Many things are possible with another level of indirection, but not necessarily worth while.

I mean, the project that started this discussion is considered "not necessarily worth while" by most people, that's why they use systems that require restarts on function redefinition instead of hot code reloading. But if you're going to the trouble of supporting hot code reloading, you ought to support (at least as an option) structure redefinition as well. Otherwise you've only gone halfway to making an actual interactive system.

This could be extremely useful in doing small logic tweaks and such. Of course it won't be able to understand your data at a high level and migrate that over.

Reply on news.ycombinator.com