Hacker News Reader

Comment by jmount | Hacker News Reader

Emacs is my primary editor, however Emacs dumper has always been a dumpster fire.

Basically they coded up so much ill-concieved and inefficient Emacs Lisp that the editor would never start up in an acceptable time. Instead of engineering around this (lazy loading services, fixing things, not doing things nobody needs) they had the great idea they could start the editor once and then core dump the in-memory state of a running editor. Then on later editor starts they would map-in the core dump and instantly be in a (somewhat) good state. Fails all kinds of smell tests and really speaks to bad taste having unbounded consequences. It is an idea that should not work and it is only happenstance that it ever worked (and it gets harder and harder as we have things like address layout randomization, file handles, and so on).

[edited "file" -> "fire", sorry! And yes I know lispers always dumped, but they are dumping the C memory environment here- not just their precious Lisp state. They should have had some appreciation for how the C environment actually worked since they decided to use it.]

View on HN

They didn't have this great idea; it came from the Lisp culture. Lisp implementations have the ability to save images. It's not described in ANSI CL, but all the "industrial strength" Lisp have this, and it is one of the means for application delivery.

If you're working on an interpreted Lisp which is getting slow to load due to its growing library, image dumping and restarting is the obvious "off the shelf" approach you already know about from being a Lisp programmer.

> Instead of engineering around this (lazy loading services

I'd be surprised if Emacs didn't have lazy loading too. Lazy loading only goes so far. Sometimes lazy loading brings in a lot of dependencies, so it is slow for those users who use whatever is being lazily loaded.

Lazy loading is a hack to work around the lack of a fast image restore.

The thing that is hacky about Emacs image saving is that it saves the C state. It's not exactly a Lisp image saving mechanism.

This is why I was abel to port this dump/undump very easily to GNU Make, which doesn't share any data structures with Emacs, let alone any Lisp stuff. Dumping the image worked just fine for the C data structures on GNU Make's heap.

It's because it's such a low-level mechanism that they are having difficulties with its portability.

I don't hear about, for instance, the CLISP project having issues with the EXT:SAVEINITMEM function due to some GLibc support changing; it doesn't rely on that.

> I'd be surprised if Emacs didn't have lazy loading too.

Not sure if this is exactly what you're referring to, but Emacs has the autoload [1] feature, which lets you declare some functions and in which elisp file to find them. Emacs won't load that file until the function is actually called. This is usually for user-called functions. Most packages use it.

[1] https://www.gnu.org/software/emacs/manual/html_node/elisp/Au...

It is an idea that should not work and it is only happenstance that it ever worked

In the 1980s it was a surprisingly common approach. For example most Unix variants had an undump command to turn a core dump into an executable. See https://www.ctan.org/tex-archive/obsolete/support/undump for example. It was also how Microsoft Office file formats were implemented until about a decade ago.

With the hindsight of experience, it is easy to say that it is a bad way to go and won't age well. But back when Emacs went this way, the problems were not as widely understood as they are now.

IIRC, "most UNIX variants" didn't have an undump command. It primarily came with a few huge applications that played these type of games. The two worst offenders that I remember were in fact GNU Emacs and TeX! A few other programs that I dealt with (porting to a new platform) which depended upon a similar hack were Sendmail and Franz Lisp.

The Sendmail "WIZ" SMTP bug was in fact caused by it's use of reloading the frozen configuration file (the pointer to the password was saved in the data section, not the BSS region that was reloaded from the frozen config, so that if you used a frozen config you effectively had an empty password).

My impression comes from the comments in the undump link that I provided.

To your list of other programs you can add Perl. It comes with a dump command to take a core dump that can be turned into an executable. I never used it as anything other than a joke, but I do know several people who independently used that feature in production back in the 1990s.

Looks like a very simple objcopy from binutils.

Maybe we just need to add an objcopy format for reading the bfdname "core" to --input-target. Sounds trivial.

In fact happened in 2001 already: https://sourceware.org/ml/binutils/2001-12/msg00360.html

Not a bad way at all, very convenient. Just ASLR might be tricky.

Now that we do have the hindsight of experience, and it is time to make that decision again, why are the maintainers going with the clearly smellier approach?

They have an immediate problem (things are slow/broken on newer glibc) that needs fixing.

Someone has posted sane code to fix the issue in a similar fashion to what they were doing before but without needing to reach into the memory manager.

The maintainer who is mad wants it fixed right (replace the whole thing and make the elisp load fast) but so far neither they nor anyone else seems to have posted code to do that.

Bird in the hand. I would hope they'd just remove the need for the mechanism totally in the future, but for now.... bird in the hand.

What would fixing it right look like?

Make elisp faster. Easier said than done. Any easy wins already happened.

Introduce various lazy loading mechanisms and switch to that by default. That doesn't help any third party plugins that people might use. In fact any such work could create ways of breaking the expectations of said plugins!

Identify/eliminate work spent on dead code paths in initial load. Again, easier said than done. And there are bound to be a lot of, "I don't see what that is useful for, but what third party plugins might need it?"

In the meantime, end users faced with the choice of a slow new release versus a fast old one, will resist upgrading. And if there is a patch that simply makes things fast, that is generally acceptable.

The current moment provides pressure to do things right. If the cheap fix is taken, there are no guarantees on when there will be another opportunity like this one. But pressures exist to take the easy out right now. And if taken, it won't be easier next time either.

I'm amazed that Chrome and Firefox load instantly and they regularly interpret scads of DOM input and complex JavaScript code.

Exactly why can't emacs do this again?

The first time you start Firefox, it reads in a whole mess of xul, html, css, and javascript. It parses it all and then dumps it to a binary file that can just be mmap'd back in next time. This is exactly like emacs, except that emacs also dumps out all of its own internal state including the state of the allocator; that makes it more fragile.

Because even with experience you don't refactor a 40+ years old piece of software easily without breaking half the ecosystem.

I'm sure there's a whole world of tradeoffs we're not seeing that the maintainers have to deal with.

It's not smellier, it's actually a great feature.

Dumping your own images will be extremely useful to many Emacs users.

This may seem strange to you, but this is actually the way many lisp systems (and other image-based development systems, like Smalltalk) work.

I build software using the commercial develop environment Allegro Common Lisp and an image dump is precisely the way I deliver software. The ultimate deliverable is an executable that loads the image and launches the main execution thread.

This works really well for us. To load up our web server takes a while due to having to load up caches etc.. so we tend to do all that once, dump the image and deploy it - cache already primed.

It had an even bigger advantage for me today. Due to some git shenanigans I somehow managed to wipe out a few days of work. No idea how... Almost resigning myself to having to recode the whole lot I just remembered that I had built an image to test just prior to attempting to check in. Clozure CL saves the source code of each function in the image. I was able to probe the image and pull out all the source code that I had just lost, saving myself a few days of tedious recoding!

Not a recommended use of images - but it sure saved my butt today!

> Due to some git shenanigans I somehow managed to wipe out a few days of work.

What happened??

I actually have no idea. I had a lot of unchecked work - about a weeks worth. (I know, I should not have let it go that far.. it was just one of those tasks that just seemed to get bigger and bigger.) It was time to check in, so I went through file by file staging it all (using Magit). I then decided to do some more testing. Built a lisp image. Then I became distracted, started thinking about home time etc... so my recollection of what I did is a little hazy. I think I saw I had some changes that I wasn't wanting to commit and test. So I think I stashed them. Then probably surfed hacker news or something.. When I got back to things I suddenly noticed that none of my changes were there any more. Nothing was staged, nothing was stashed. It was all gone. I spent a few hours looking around for it but all to no avail.

It is the second time I have lost work by misusing git. I think the other time I was messing with rebasing and branching and all my work disappeared.

I do really need to learn how to use git properly. But in the meantime I am not taking any more chances. I have set up a zone on our internal SmartOS server which uses ZFS. An hourly cron job on my machine rsyncs my source directory to the zone and then creates a ZFS snapshot. So, hopefully no matter how badly I use git now I will never be able to lose more than about an hours worth of work.

> so I went through file by file staging it all

When you stage a file in git, what it actually does is to create in its object database a blob with the contents of the staged file, and then it points the corresponding entry in the working tree index to the blob's hash. The blob object is only removed during the periodic garbage collection, and only after a configurable period of time since its creation has passed.

Therefore, once you stage a change, git will keep a copy of it for at least two weeks unless configured otherwise. Even if nothing points to it anymore (due to, for instance, an errant git reset --hard), there are ways to find its object hash (something like git fsck --unreachable, or even a ls -lR in the object database directory), and once you have the hash, you can use git cat-file to retrieve its contents.

Result! git fsck --unreachable has pulled up the stray commit. There it all is, right in front of me. Incredible! Thanks.

Seriously? Wow! I will try that first thing tomorrow. Thanks.

Look up git reflog or so.

Recovering from a lisp image is pretty darned cool. :D

> I had a lot of unchecked work - about a weeks worth > ... It is the second time I have lost work by misusing git

I realize this is _off topic_, but one thing that helps me a lot is making regular small commits (that describe some small idea of the change), and then rebase them later once I get all the linting / tests fixed. (Obviously not on master ;)) It's easy to go too-granular here, but it's very handy to be able to reorder the commits, or squash "fixup" commits, to make things easier to review.

I spend more time rebasing than I probably need to, but on the other hand I've never lost work. ;)

For example (adapted from a PR I was working on this morning ;))

    git commit -m "Add Clone Foo feature"
    git commit -m "Reorganize tests"
    git commit -m "Fix indentation"
    git commit -m "Add generator for Baz items"
    git commit -m "Add test of Clone Foo feature"
    git commit -m "fixup linting in application"
    git commit -m "fixup test linting"
    git commit -m "fix Foo test to use correct selector"

    git rebase --interactive master
    # reorder fixup commits, squash them together with the things they fix

You can always rebase and squish all of these down into one commit later before you make your pull request (or, leave them as-is if your team is OK with un-squished PRs).

> # reorder fixup commits, squash them together with the things they fix

The --autosquash option of git rebase can save you some work:

       --autosquash, --no-autosquash
           When the commit log message begins with "squash! ..." (or "fixup! ..."), and there is a commit whose title begins with the same ..., automatically modify the todo list of rebase -i so that the
           commit marked for squashing comes right after the commit to be modified, and change the action of the moved commit from pick to squash (or fixup). Ignores subsequent "fixup! " or "squash! " after
           the first, in case you referred to an earlier fixup/squash with git commit --fixup/--squash.

           This option is only valid when the --interactive option is used.

           If the --autosquash option is enabled by default using the configuration variable rebase.autoSquash, this option can be used to override and disable this setting.

(Not off-topic for me. :-)) I will have a play with all this. Now that I have my source being backed up to zfs I can be a bit more comfortable with experimenting with new workflows.

Thanks

To help learn to use Git properly, start by making many frequent commits from the command line throughout the course of the day. I would say that an appropriate pace is about once every 30 minutes.

When you're done with the feature, then depending on your preference, you can either merge your branch into the master branch, or you can rebase your changes onto the master branch (my recommendation). In both approaches you may wish to squash your commits, so that you ship perhaps just one commit or fewer than your full set of working commits. Then you prepare the changes for code review and for pushing upstream.

The best way to learn about and become comfortable with a tool is to use it regularly. As you approach mastery it will become a swiss army knife. Frequent commits give you a lot of utility for the same reason that editor undo/redo buffer does, except it's persistent.

If you are worried about screwing things up by running the wrong git commands, then check out git's reflog feature, and how you can use `git reset --hard` to roll back changes via the reflog. Virtually every change you make to your local repository is versioned, and the reflog shows you that history and allows you to roll back.

My first step with git is always creating a new branch, then I commit multiple times a day with "wip" or minor notes as the commit message. It is purely to capture the stream-of-consciousness of the code. When it comes time to create a sensible patch I immediately branch again. The old WIP branch is not touched until I've completely merged the feature. On the new branch I usually reset then commit hunks in some way that makes sense, or if the changes are small squash everything into a single commit.

You can also do the same thing with tags if you like.

There is no penalty to branches in git... use them. Frequently.

I got into trouble at one stage mixing branching with rebasing. This really put me off branching. Creating a new branch to perform the merge is a stroke of genius - thanks! I will be doing that right away.

Thanks for the explanation! I am not a heavy user of git stash, so not sure if git reflog could have helped you, but I am grateful that somebody steered me toward git reflog early in my learning git. With reflog and frequent commits, it's very hard to lose work, short of deleting the entire project or its .git/ directory. rsync+ZFS also sounds like nice combo for running around git in order to prevent future mistakes destroying work.

> An hourly cron job on my machine rsyncs my source directory to the zone and then creates a ZFS snapshot. So, hopefully no matter how badly I use git now I will never be able to lose more than about an hours worth of work.

This is a thread about Emacs - no need for ZFS, Emacs can already do versioned file backups for you!

http://stackoverflow.com/questions/151945/how-do-i-control-h...

That is very interesting. I do have Emacs making some sort of backup, but have never managed to get it to work reliably. I'll follow those instructions.

What was confusing for me is understanding that by default Emacs does not back up on every save. It makes a copy when you first open a file, so only the version of the file before you start editing it is what gets backed up.

What sort of software do you develop? By that I mean what field are you in?

Not a lot of commercial lisp projects make the news that I read.

Well, for instance, this project on which you are reading news right now is one...

That'd be why I said "not a lot of" and didn't say "none"

I agree that dumping an image is not a great idea in the modern world, but the concept(1) dates right back to when emacs was still based on TECO on the PDP-10: http://pdp-10.trailing-edge.com/mit_emacs_170_teco_1220/01/i... (and I suspect the available CPU horsepower back then was such that you pretty much needed to do tricks like this to get quick startup even if you were careful about how much lisp code you ran at startup).

(1) as a part of emacs, that is -- I'd guess that "language runtime takes an image of its pre-initialized state and then just loads that" was used in non-emacs contexts before that.

The first Lisp from the early 60s did already dump images.

Did it dump Lisp images, or the image of the entire process (including all the non-Lisp stuff)?

It appears that all the complexity and instability here comes from the latter part. Lisp itself was always designed around this image dumping concept. But C and its standard library were not.

This is Lisp on the IBM 704/7090 and PDP-1 in 1960-1964. Ritchie would not start work on C until 1969 and there was no standard C library (which, btw, is completely irrelevant to the runtime state of a process) for decades. There were no processes or virtual memory (the 7090 didn't even have overlays) or non-Lisp stuff other than the punch card/paper tape loader.

Actually, many early unixes included undump, which let you build an executable from a coredump.

And what were the stability guarantees for such things? Could you expect to take such a dump, bring it to a different system (potentially with different hardware, different minor OS revision etc), and expect it to work?

In those days, you could pretty much never take any software to different hardware, and even different minor OS revision was asking for trouble. What you are asking for isn't something anyone expected back then. Different hardware was _really_ different.

Right, that's kinda what I was expecting. But that also explains why it was a mistake to adopt this model for Emacs, or at least to stick with it for so long (to clarify, I specifically mean full process dumps, not Lisp world dumps) - it's a different world, and it has been different for a very long time, with software and OS/hardware significantly more decoupled.

> it's a different world, and it has been different for a very long time, with software and OS/hardware significantly more decoupled.

This argument makes about as much sense for core files as it does for binary executables, which is to say, not very much.

Why? A binary executable does not rely on the very fine details of the implementation of the OS heap manager, for example, while a dump necessarily would.

> A binary executable does not rely on the very fine details of the implementation of the OS heap manager, for example, while a dump necessarily would.

What are you talking about? What is it about brk(2)/sbrk(2)/mmap(2) that would be a problem?

Correct me if I'm wrong, but if you make a process dump of the entire process, that dump would include internal structures used to maintain its heap, no? If those structures change between OS versions, for example, then the process dump from an older version would represent corrupted heap when loaded on a newer version.

None of the kernel data structures are saved or even visible to the process - the kernel memory (on Linux x64 everything mapped in the upper 128TiB) is not readable by the process.

You can have a problem if the kernel system call interface changes, but that is a problem for every executable. You would need to re-compile everything on the system.

Are all heap-related structures be stored in kernel memory, though? There's nothing in userspace for bookkeeping?

The kernel just gives you pointers and sizes via mmap/sbrk, it is up to you to keep track of them.

> Instead of engineering around this (lazy loading services, fixing things, not doing things nobody needs) they had the great idea they could start the editor once and then core dump the in-memory state of a running editor. Then on later editor starts they would map-in the core dump and instantly be in a (somewhat) good state. Fails all kinds of smell tests and really speaks to bad taste having unbounded consequences.

How else would you compile a dynamic programming language program into a binary executable?

BTW, something that just came to mind: this "bad taste having unbounded consequences" of copying the entire process memory is exactly how fork(2) works.

Doing something that relies on OS internals is bad taste when done in userspace.

Interface stability aside, what fork does is simpler: a copied process can rely on its environment matching its parent's enough for a large subset of the contents of the duplicated address space to retain their meaning. Serializing a process's address space's contents and then loading it into an "unrelated" process later makes it more complicated to know what memory contents can still be relied on (e.g. ASLR, file handles, etc.).

Nothing of what Emacs or Common Lisp systems or Smalltalk systems do when they dump images relies on OS internals.

Emacs unexec relied on glibc internals but the new portable dumper does away with that and only relies on gasp Emacs internals.

Where do you people come up with that stuff?

> Interface stability aside, what fork does is simpler: a copied process can rely on its environment matching its parent's enough for a large subset of the contents of the duplicated address space to retain their meaning. Serializing a process's address space's contents and then loading it into an "unrelated" process later makes it more complicated to know what memory contents can still be relied on (e.g. ASLR, file handles, etc.).

Everything you mentioned except for ASLR is a problem for forked processes, and it boils down to kernel state (file descriptors and signals being two that I can think of right now).

Yes, I remember building Emacs in the late 1980's and watching the final dump. I guess 30 years later hardware hasn't improved enough to avoid this? People are building editors in JavaScript today. Perhaps the Emacs Lisp engine needs to be tuned a little more? That Guile Emacs has been decades in the making...

Sounds like it needs the neovim treatment: neomacs.

Once upon a time this basically happened: xemacs

God a neomacs written in a faster, more modern lisp would be a dream.

It's called guile emacs.

https://lwn.net/Articles/615220/

https://www.emacswiki.org/emacs/GuileEmacs

Guile Emacs isn't a bottom-up rewrite of Emacs [1], but a project to replace the elisp runtime with Guile. One important thing it does not do is get rid of/convert any ELisp, because of how much is invested in it: 71% of the ~2M LOC codebase is ELisp: https://www.openhub.net/p/emacs

Guile would run the ELisp code. What's holding the project back, IIRC, is that it doesn't yet do so perfectly.

[1] which is something I (and likely most users) frequently think it sorely needs, until we realize A) the monumental size of the task and B) how much refinement the current version already has, as per http://www.joelonsoftware.com/articles/fog0000000069.html

Interesting. Any idea on why it's not got more traction? Or is it actually on track to become the canonical emacs?

Reading that LWN article, it looks like Guile wasn't a great choice from a community perspective, but more than that, by working internally to the emacs project it's subjected itself to internal standards of interoperability that it'll never really be able to hit (where a pseudo-hostile fork like xEmacs was might have sufficient momentum to force some accommodation).

> Reading that LWN article, it looks like Guile wasn't a great choice from a community perspective

I think going forward it will be. Between Guile-Emacs, Guix, and GNU Shepherd you have the best-supported Lisp Machine operating system analogue available right now: https://www.gnu.org/software/guix/

I am excited about GuixSD and I think a lot of other people will be.

No it wont, at least as far as Emacs is concerned.

Nobody is stepping up to do the work, plus Guile has serious bugs on Windows and OSX, single-digit number of developers and few, if any, users. So Guile Emacs is really a pipe dream that people like to bring up from time to time.

> Nobody is stepping up to do the work, plus Guile has serious bugs on Windows and OSX, single-digit number of developers and few, if any, users. So Guile Emacs is really a pipe dream that people like to bring up from time to time.

You forgot to mention that BSD is dying.

This state of affairs is different from any other Lisp implementation how? And why would it stop progress from being made?

I've been happily running SBCL and CCL on Linux and OSX for years without issues. CCL on Windows too.

Guile has 1-2 people working on it part-time, and Windows/OSX are not a priority because:

+ GNU project, duh.

+ Nobody in Guile-land cares about Windows/OSX enough to step up and fix issues.

+ Even if they did, Stallman would tell them not to.

So the state of affairs is indeed very different from the CL Lisp implementations. Not to mention, Guile has pretty much no userbase to speak of. There are commercial entities releasing products with SBCL and CCL in addition to the very healthy opensource community.

> best-supported Lisp Machine operating system analogue

Hardly. GNU is far from anything Lisp Machine. The Lisp Machine was never a system hacked up by scripts running on top of some Unix copy with a fancy name.

It sorta works, but it's hard to keep it working, so people don't really do much with it.

Shockingly, the compatability was pretty good, last I checked.

If you really want an emacs written in a sane Lisp, and don't mind losing gnuemacs compatability, check out Hemlock. or Edwin.

> check out Hemlock. or Edwin.

Or Climacs https://www.common-lisp.net/project/climacs/

It seems there's a rewrite as well: https://github.com/robert-strandh/Second-Climacs

Oh. I forgot Climacs.

Well, there you go. Another CL editor.

Here is another one:

https://github.com/cxxxr/lem

As I note elsethread, Guile Scheme is not a 'faster, more modern Lisp,' because Scheme is not Lisp. Scheme is a Lisp-like language, which does have a place in the world, but it's not Lisp, and it's not (IMHO) well-suited for large software projects like emacs.

I personally oppose a Scheme-based emacs because, having moved from C to a more Lisp-like language, I don't think there would be the will to actually move to real Lisp. You might accuse me of letting the best be the enemy of the good (which is a fair criticism, both of my position re. porting emacs & of Zaretskii's re. the dumper), but part of why I use emacs and Lisp is that they are not good: they are the best.

Emacs-Lisp is dynamically scoped. That's precisely the kind of mistake (yes, mistake), that hinders large scale anything.

And who cares Scheme is not Lisp? It's close, and it's reputedly cleaner. Besides, there are many Lisp dialects, some of them just as different from one another than Scheme is from them.

> Emacs-Lisp is dynamically scoped. That's precisely the kind of mistake (yes, mistake), that hinders large scale anything.

Yup, default-dynamic scope is a mistake, although in the particular case of emacs it has made certain code very clean (and led to plenty of bugs, as well). Emacs has added lexical scoping, which is a start.

Common Lisp's lexical default and optional dynamic scoping provides the best of both worlds.

> And who cares Scheme is not Lisp? It's close, and it's reputedly cleaner.

Among other things, its continuations are semantically broken, preventing correct implementation of UNWIND-PROTECT.

It's a good language for its intended purpose (teaching & research), but the language as standarised is not well-suited for building large projects (individual implementations, of course, can be quite good — but then one enters the world of implementation-dependence).

Common Lisp is superior for large projects, in part because it's much more fully-specified, in part because that specification includes more features, in part culturally (because Lisp systems have more often tended to be large and long-lasting).

In TXR Lisp, a Common-Lisp-like language which has (delimited) continuations, I came up with a pragmatic approach w.r.t. unwind-protect which works well.

I introduced "absconding" operators which allow a context to be abandoned (via a non-local exit) without invoking unwinding. Absconding just performs the jump to the exit point, without doing any clean up along the way. That's it!

The idea is basically: why, when temporarily leaving a continuation, should we clean up resources, when we intend to resume the continuation and want those resources intact? If it should happen that we don't resume, so what; garbage collection will just have to take care of it. We wouldn't do that with threads, or coroutines, right? When a thread suspends, we don't close its files for fear that it might never wake up to use them and close them.

With this absconding, I implemented a workable yield construct. I can have code which, say, recursively traverses some structure and yields items to a controlling routine. That recursion can have exception handling and unwinding which works normally, as if the yields were not there. When the recursion performs a normal return, or a throw, all the unwinding takes place normally. The yields use absconding, and so they don't disturb anything. When a continuation is resumed, all of the dynamic handlers are in place, including the unwind cleanups that were never touched. Nothing was torn down.

Also in place are dynamically scoped variables. TXR Lisp's continuations play nicely with those also.

If you compare this brilliantly simple solution to concoctions involving dynamic-wind, dynamic-wind looks hopelessly silly, like what were they thinking when they came up with it?

Just don't unwind when you temporarily leave a context. The sky does not fall. Your CS degree is still hanging the wall. Everything is cool.

It was called XEmacs.

As for the more higher level stuff, spacemacs does a great job of that. It really makes configuration files, separation of concerns, vim bindings, and many other things sensible and user friendly. It doesn't touch the lower level code though, as it is just an elisp layer on top of emacs itself.

> Emacs dumper has always been a dumpster fire.

This made me laugh. Thanks!

Reply on news.ycombinator.com