Continued progress porting Emacs to Rust | Hacker News Reader

Continued progress porting Emacs to Rust

db48x.net

163 pointsealhad58 comments

To anyone wondering why this might be a good idea, there are some interesting comments in a GitHub issue (https://github.com/Wilfred/remacs/issues/305):

> For users, I hope Remacs will be faster (it's easier to optimise), more robust (we have stronger type checks and the ability to add unit tests), better documented (see #262) and somewhat more featureful (we have some ideas for solving the dumper problem that would allow users to dump their instance as an image) than GNU Emacs.

> I think we will get problems when the current generation of emacs devs retire and there's a better chance that emacs will survive if we try to make it more appealing to young developers. Rust seems to be a good option.

> For users, I hope Remacs will be faster

The main reason my emacs is painfully slow is that most elisp code blocks. Stuff like hitting tab for auto-completion might perform a blocking query from a language server hanging emacs for multiple seconds. Opening a file tries to start a language server and blocks the editor in the process. Etc.

Making emacs itself faster won't make the experience of working with emacs any better.

> The main reason my emacs is painfully slow is that most elisp code blocks.

Note that multithreading is now in Emacs master, and some core packages (e.g., Tramp, which used to block when opening a remote file) have already been updated to make use of it, so starting with the next major release things will be improving on this front.

Some things like indenting a large file or even searching in a huge buffer can be quite slow. For instance I just tried indenting a ~1000line C buffer and it took about 2 seconds. When dealing with very large files (hundreds of MB) some basic movement and editing commands can be quite slow. Dealing with very large TAGS file (for the linux kernel for instance) is annoyingly laggy as well.

I agree that it's not the panacea though, Emacs would benefit from using a more threaded or event-oriented programming model for many things.

Emacs does block on many operations, but I've never had it be really noticeable on tab-completion type stuff.

Though I only use auto-complete on emacs commands, so maybe I've just missed it.

It depends quite a lot on what kind of autocompletion you're using, for which language. It's a mixed bag.

An even better chance is that this project will turn those young developers into old ones.

> make it more appealing to young developers

In that case, they should rewrite the Lisp code in JavaScript.

Yes, I'm being sarcastic.

The author misunderstands some parts of the C language, in particular related to static variables. "In fact, these aren't just file globals accessible to only one translation unit, these are static variables that are accessible across the whole program." gets the meaning of C's static exactly backwards: C file-scope static variables are only accessible (by name) in their translation unit; all other file-scope variables are extern by default.

More importantly, they write:

    static struct Lisp_Objfwd o_fwd;

"... creates an (uninitialized) static variable called o_fwd". No. C static variables are implicitly initialized to zero of the appropriate type, in this case, apparently a struct containing only a null pointer. This also means that their

        unsafe {
            #[allow(const_err)]
            static mut o_fwd: ::hacks::Hack<::data::Lisp_Objfwd> =
                unsafe { ::hacks::Hack::uninitialized() };

"a lot more typing to get a proper uninitialized value in a Rust program" is just useless voodoo.

You are entirely correct, and I should perhaps have been more precise. However, because Lisp_Objfwd is basically a pointer, zero-initialized is pretty much as good as uninitialized. It's not a useful value.

It fails a NULL check, where an uninitialized value (probably) won’t. That seems like a pretty important distinction, no?

In C, yes, that's a useful distinction. One of the goals of Rust is that you'll never have to check that manually. You'll never encounter a pointer that points to an invalid, uninitialized, or unallocated object. Since we're writing some unsafe code here, we can operate outside that restriction as long as we promise to put things right before the end of the unsafe block. This can be simplified somewhat once everything is written in Rust.

The C spec says zero in a pointer context is null (and the other way around)

Can you elaborate on “as good as uninitialized”?

You can't ever usefully dereference a null pointer, so the only thing you can safely do with it is check to see if it is null. I commented on a sibling comment in more detail, but basically in Rust no reference is allowed to be null except temporarily inside an unsafe context.

There isn't anything you can do safely with an actually uninitialized value, though.

In the internals of TXR Lisp, I used a null pointer to represent the symbol nil, with all the useful semantics that follows. If an object containing Lisp values is initialized to zero, those values are nil.

Yes, Emacs and Remacs do the same thing. All Lisp values are tagged pointers, and a nil gets encoded as a zero value with a zero tag. It's simple and easy to check for, and you won't have any problems as long as you always remember to do so.

However, this is not a Lisp value; it's a pointer to one. Specifically, it's a pointer to memory set aside to store a specific Lisp variable's value. Leaving it zeroed makes no sense, because eventually someone is going to store a value in there and it'll have to be allocated anyway. Better to do that all in one large block, rather than a thousand tiny ones. Even if we wanted to lazily allocate these, we can't leave a simple Rust reference (or pointer) null past the end of the unsafe block.

Rust does have an Option type, and if you use it on a type that isn't nullable, then the compiler will use the zero value for None variant and all other values for the Some. Since references can never be null, an Option<&T> will always take up the same amount of space as a &T; there's no additional overhead. At some point, when enough of the C code is ported to Rust, I'm sure we'll start using this to represent all Lisp values and a Lisp nil will always be a Rust None.

> Even if we wanted to lazily allocate these, we can't leave a simple Rust reference (or pointer) null past the end of the unsafe block.

So make it an Option, and get rid of the unsafe block altogether.

Perhaps one day we will be able to do so.

> gets the meaning of C's static exactly backwards

Which one? It has multiple.

First, there is storage duration:

C99 6.2.4 "There are three storage durations: static, automatic, and allocated."

Variables with external linkage defined at file scope are static in the sense that they have static storage duration; it is not incorrect to call them static variables, though a bit unusual. If they have external linkage, it's more common just to call them globals.

The static storage class specifier keyword used at file scope causes a name to have internal linkage. In a block scope, it requests the above static storage duration.

> Which one?

The one that was entirely clear from context: Linkage.

When I first heard of this project I thought "Well, that's insane, it will never be finished, and will surely be abandoned in six months". Glad to hear that it's making good progress! I love it when I'm wrong about stuff.

Remember that most of Emacs is written in Emacs Lisp. If you take the entire source code distribution, around 15% of it is written in C. Those 15% are of course fairly complex, however it's at least less code than you may think.

Yes. The last time I looked, it was about a million lines of elisp, and only about a hundred thousand of C. We'll have it cleaned up in no time.

    DEFUN ("atan", Fatan, Satan, 1, 2, 0,

Hah I couldn't read that with a straight face!

Agreed! It's a great example.

That's beautiful!

And there are related efforts to upgrade the lisp used in emacs to a more modern and standardized version.

I think it is GuileEmacs I am remembering.

I hope to two efforts are compatible.

That's the first thing that crossed my mind when I saw this project but I think it might be a mistake. Convincing people (and contributors) to ditch C-emacs for "remacs" will be hard, convincing people to ditch elisp-emacs for guilemacs will be hard but doing both at the same time might be just too much.

If remacs manages to reach the point where you can use it to replace emacs in-place without any breakage while bringing better performance and more stability it will be a tremendous success IMO.

Although Scheme has a standard, I wouldn’t really call Guile ‘more modern and standardised’: Guile has no standard (it extends the Scheme standard in incompatible ways), and ‘modern’ is hard to define. Emacs Lisp added lexical scope in the past couple of years, probably more recently than Guile added a new language (as opposed to library) feature — is that modern enough?

I wouldn’t imagine that the two efforts are compatible, because they are both efforts to rewrite the Emacs Lisp engine in another language, one in Rust & one in Guile Scheme. It’s possible that one might be able to call over to the other, through its CFFI.

I’d like to see Emacs rewritten in Common Lisp, and so I’d prefer a Rust engine to a Guile engine: Rust is a static language, and a Rust engine is unlikely to lead to Rust-extensible Emacs code, while a Guile engine is likelier to lead to Guile-extensible Emacs code, which is completely the wrong direction IMHO.

it extends the Scheme standard in incompatible ways

What does "incompatible" mean here? Incompatible with the Scheme language reports, or incompatible with other Schemes' extensions?

Guile includes a bunch of functions that aren't part of the standard, and have been around before Scheme's standard thought to implement libraries.

It’s a mistake to replace emacs lisp. For a start everyone’s custom code and configurations would have to be modified or scrapped. Package libraries would have to be ported. Much better to just improve it in a backwards compatible way or with libraries. Both of which are happening.

The goal of Guile Emacs is not to replace Elisp; it is to replace the implementation with Guile.

My thinking as well. This is a golden opportunity to replace Elisp.

Edit: Guile won't be used, because of the burden of contributing. https://github.com/Wilfred/remacs/issues/66#issuecomment-273...

This is very unfortunate. Emacs itself also requires copyright assignment, though, so if Remacs doesn't then it's virtually impossible for GNU to switch code bases (not that it would have been likely anyway).

Remacs’s README is really well written. More projects should try this hard to sell their ideas. This totally beats the GNU Emacs website at selling me on why I should use Emacs.

Emacs rather famously has a build process that involves first compiling a minimal elisp environment (written in C and maybe some assembly) that then loads all the editing routines (which are written in elisp) and then dumps itself out as a binary (to save the load time, which is substantial, on subsequent start-ups).

I assume the Rust-based emacs would do the same thing?

We still use the same unexec process as normal Lisp, not because we really like it but because it's easier to make small changes. It really is a defect attractor though (I even introduced one a few months back, and had to debug it), so we do want to simplify it.

not sure, but I guess that process is to make sure Emacs can be built from scratch, ie without relying on proprietary software.

The build process of Emacs doesn't rely on any proprietary software. But it is quite complex :)

Not an Emacs user, so forgive me if I'm being dense, but is this (from the README) really true?

  Emacs will change how you think about programming.

  Emacs is an incremental programming environment. There's no 
  edit-compile-run cycle. There isn't even an edit-run cycle. You can 
  execute snippets of code and gradually turn them into a finished 
  project. There's no distinction between your editor and your 
  interpreter.

I'm assuming what it means is "Emacs will change how you think about programming Emacs Lisp". Which is a rather narrower proposition, and much less of an incentive to switch, since I doubt many non-Emacs users are particularly interested in programming Emacs Lisp.

Yeah, it does that. Its not just emacs lisp, clojure, clojurescript and CL2 all have tight integration.

The difference is what it means for your repl. You can trivially attach your repl to a running process. So you change an expression, tell emacs that you'd like that to be sent to the repl, and because the repl is attached to your running process everything updates immediately.

I put this together years ago: https://www.youtube.com/watch?v=W2NJppeLsN8

It does a pretty terrible job of demonstrating what this might mean for something like web development where the feedback circle is usually as bad as it can be.

Clojure + Cider inside of "Spacemacs" is the most enjoyable development environment I've used in over 20+ years of programming. It's amazing how quickly you can get stuff done.

Lisp changes how you think about programming. Emacs gives you the shortest path to a code/test feedback loop.

Yes, it’s true — and it’s not just true of elisp (which is a decent language) but also of Common Lisp, Python, Scheme and others. It’s probably most true of elisp, because the editor itself is written in elisp, but one gets something of the same tight edit-execute cycle in those other languages too.

Once you’ve been there, you’ll probably never want to go back. My only wish is that emacs were written in an even better language than elisp (like Common Lisp).

> My only wish is that emacs were written in an even better language than elisp (like Common Lisp)

There are a lot of bad things in CL too, unfortunately. I wonder if a Lisp will become really popular in the near future. Maybe Racket?

There are 'bad things' in any programming language.

Common Lisp is actually quite a bit more advanced out of the box than Emacs Lisp: lexical binding, excellent error handling, good support for compilation, extensive object-oriented features, it has multiple implementations with independet code bases, implementations with threading, ...

CL has been used to develop Emacs editors already.

But the thing is a non-starter. There are a few million lines of Emacs Lisp for a specific editor, which nobody will rewrite.

Emacs Lisp has lexical binding these days.

I’d like to see an Emacs with a Common Lisp core, and an elisp execution engine to run all the legacy elisp code. https://github.com/blindglobe/clocc/blob/master/src/cllib/el... looks like a decent first start, although it supports an older version of elisp.

Someday if I have the free time I’d love to spend time working on such a thing. Maybe when I’m retired!

> Emacs Lisp has lexical binding these days

sure, but you need to support legacy software, too. Which is most Emacs Lisp ever written.

Anyone who has seriously proposed porting Emacs to any language will have made plans to support running the existing Elisp code without modification. You can easily write an Elisp compiler in Common Lisp.

In fact, the original Lisp machine eventually supported Common Lisp as well as the original Lisp Machine Lisp. You could write any individual function in either language, and call from one to the other transparently.

> You can easily write an Elisp compiler in Common Lisp

I have no idea how easy it is and I have seen no attempt to do that (running the existing Elisp code without modification on top of a CL runtime). If it would be easy, I'd expect it to be done already.

The https://www.cliki.net/CL-Emacs page seems to of no help...

Do you know of any attempt demonstrating how easy it actually is?

> In fact, the original Lisp machine eventually supported Common Lisp as well as the original Lisp Machine Lisp. You could write any individual function in either language, and call from one to the other transparently.

That was a major engineering effort - putting both languages side by side onto a single runtime.

Well, I grant you that it won't be done in a day.

Of course CL is more advanced than Emacs Lisp, it's a very interesting platform! I have grown tired of some of its aspects though, like those endless lists of possible keyword arguments.

But this is just me nitpicking about elegance.

Richard Stallman doesn't like that either. There is no support from him for a CL version. There never was. It's nothing the core developers of GNU Emacs are interested in.

But if we look around - most alternatives have their variants of keyword alternative mechanisms.

Racket: https://docs.racket-lang.org/guide/lambda.html

Guile: https://www.gnu.org/software/guile/manual/html_node/Coding-W...

Chicken Scheme: https://wiki.call-cc.org/man/4/Extensions%20to%20the%20stand...

My interpretation is that it's not a statement about writing code but running it. It's accurate IMO but probably not terribly important in the grand scheme of things.

> I doubt many non-Emacs users are particularly interested in programming Emacs Lisp

By definition, if you're programming Emacs Lisp, you're an Emacs user. Emacs is the interpreter.

> By definition, if you're programming Emacs Lisp, you're an Emacs user. Emacs is the interpreter.

That's not necessarily true. Guile supports Elisp as a language. You can write Elisp sans Emacs.

The more usual statement is that learning Lisp will change how you think about programming. Many have said that it will expand your frame of reference, and make you a better programmer in whatever other languages you end up using. I agree with them, and also that having a programming language so easily accessible inside your editor is also similarly liberating. You can decide that your editor can work more efficiently in some way, then take a few minutes to extend how your editor works, and then continue on with your original task.

As languages go, I would say that Forth and Prolog also have this quality of expanding your frame of reference.

It is true. Learning any Lisp changes how you think about programming. Learning Emacs Lisp will make you wish all development was like that.

Neovim vs Remacs now.

Reply on news.ycombinator.com