State of Emacs Lisp on Guile
emacsninja.com
emacsninja.com
Have everything under Guile umbrella? Remember, Guile is getting trickier and trickier to build over the years and I don't want that from scripting language whose main idea was to embed it (presumably also ship the source alongside your program in some cases). I'm getting impression even python source & build is easier to ship and you can't go below that.
Also, modern Guile compilation time is abysmal. I recently ditched Guix [2] from my system because bootstraping on fairly modern laptop would take hours, mainly because Guile is trying to compile some helper scheme libraries needed for guix to work and recompile everything again with 2-3 different gcc versions (which are bootstraped too).
Here is what Andy Wingo had to say on the subject[1]:
Guile can implement Emacs Lisp better than Emacs can.
We can compile Emacs Lisp to Guile's VM, and make C shims to present a suitably compatible C interface to the rest of Emacs' C primitives.
No one will notice! Except that after a switch, Emacs would be faster, more powerful, and have the ability to access all of Guile's facilities -- the Scheme language, other languages implemented for Guile (Javascript, Lua, ...), a proper ffi, dynamically loadable libraries, a module system, the numeric tower (rationals, bignums, etc), Guile's existing libraries, delimited continuations (!), fast bytevector access, native threads, etc.
On a language level, I am convinced that not only can Guile implement Emacs Lisp, but it can provide a coherent story for tight interaction between e.g. minor modes in Javascript and Scheme and Elisp, etc. More details will come later, though feel free to bring up specific points.
There have been some suggestions on the list that "Guile people want to rewrite Emacs in Scheme". Though Scheme does have some facilities to offer, personally I do not believe in rewriting software. I've failed enough times, thank-you-very-much -- and if we could retrofit a compiler into Guile in two years via correctness-preserving transformations, surely we can replace the implementation of Elisp without causing problems.
My goal is to make it the obvious decision for the Emacs maintainers to say, "yes, let's switch to Guile's implementation of elisp, because it will make Emacs better and have no drawbacks". We're not there yet -- and that's why I wanted to put off this mail -- but we'll get there.
[1] - https://lists.gnu.org/archive/html/emacs-devel/2010-04/msg00...
Absolutely nothing of what he described in that email materialized. And it's no wonder it didn't, since a lot of it is simply wishful thinking. In addition, he didn't address the fact that there are political and not technical reasons behind the absence of certain features (e.g. FFI). They could be done today, without Guile, assuming a consensus on the political front existed.
On the other hand, Emacs Lisp has gotten a lot better and not only knocked out some of the features mentioned by Andy (dynamic modules, native threads) but is also making progress towards new directions (GCCEmacs).
And as I described in my other comment, I completely disagree with his premise that having Emacs be extended in JavaScript, Lua etc would be a plus. The last thing you do with a cohesive ecosystem like Emacs Lisp's is to introduce needless fragmentation that takes away its core strengths.
I also disagree, but that boat sailed long ago... For years now Emacs has allowed dynamic module loading, which lets Emacs be extended by C or anything that compiles down to C (which is pretty much any language). So it can already be done.
What Guile will offer is in some cases skipping the module compilation step, and allow more direct access to languages which can be used through the Guile interpreter.
Furthermore, whether you or I like the idea of extending Emacs through Javascript or Lua, there are plenty of other Emacs users who do, and I say let them at it!
Emacs needs more new users, and if some can be attracted through compatibility with their favorite language, I welcome them.
I'd personally like to keep Emacs pure and Lispy, but that's already not true and it's very unlikely we're going to go back.
That's not to mention that significant parts of core Emacs has been written in C for decades now. It'd be much nicer if some of that C code could be rewritten in Scheme.
Going through C is not a counterargument, since it's doesn't compete with interactive development in Emacs Lisp. The technical drawbacks are many and for that very reason you see no fragmentation in the ecosystem today.
Being able to write a Lua or JavaScript function on the fly, inside Emacs, and having Guile execute it would make these languages direct competitors to Emacs Lisp. You would expect to see newcomers take the easy way out and never bother with Emacs Lisp. Fragmentation. Dilution. That is the core issue.
You seem to think that these sort of sacrifices should be made in the interest of popularity. I don't agree at all. Emacs has no reason to make populist compromises. It has certain qualities that have stood the test of time and have kept it extremely powerful. To go on the populist road would ensure that Emacs becomes just like everything else out there. A mediocrity.
Anyway, I don't see this as the primary benefit of Guile Emacs.. the rest benefits that Andy Wingo mentions are much more interesting and valuable to me, not JS/Lua compatibility.
I'd also much prefer to write my own Emacs code in Scheme than eLisp (though, again, Guile Emacs does not entail replacing eLisp packages with Scheme -- you can still use eLisp if you want, and existing eLisp packages will still work).
I wouldn't mind Scheme as an alternative, the only alternative, except I don't think I'd use it myself since I would still see Emacs Lisp as domain-superior.
Agreed on the rest, except as I said, none of that materialized from the Guile camp whilst we had plenty of that goodness emerge in core Emacs.
Having packages in multiple languages hurts the ability of people to help contribute to each others' work (especially if they don't have time or space to learn multiple languages).
However, it seems that some people have another goal in mind with the Guile-based Elisp: actually replacing Emacs Lisp, or phasing it out, or whatever. I think that is a terrible idea. I also think their goal of mixing languages inside of Emacs would have the negative consequences you describe.
The problem is not technical, but political. For years, RMS has opposed an Emacs FFI due to his belief that it could be used to subvert Emacs goals by making it possible to do proprietary extensions.
Emacs Lisp fits the problem domain like a glove. Someone in this thread called it 'the worst Lisp' but that's a mischaracterization. It's actually a good, practical, Lisp and far from the worst. In short, there is enormous value in the current Emacs Lisp ecosystem, and every heavy Emacs user knows it.
Guile doesn't really have much to offer, given its terrible performance, lack of interest from developers (and some would say users), platform-compatibility issues and strong potential for diluting a very valuable, very cohesive ecosystem.
It is a project that is slowly languishing while Emacs Lisp is slowly but surely getting better.
It is wrong to say that Guile has terrible performance. Guile Scheme is more performant than the standard implementations of elisp, python, ruby, and other so-called scripting languages. Each major release of Guile has had significant performance improvements, most recently a JIT compiler. It is true that the elisp implementation on Guile is slow, but that's a reflection of the project being a prototype and not production-ready. It could be fast if there was interest in developing it, but there isn't so it's just another interesting but dead project.
But it is a fairly clunky language with all it's let, letstar, letdash, letstardash... You know what I mean. I am no elisp history expert, but I believe that is due to it's age and how easy it is to extend.
And just to reiterate, I have never felt limited by it, I really like it, I would not change it and it solves the domain problem just fine.
On the other hand languages like TXR Lisp and Emacs Lisp have made certain compromises in order to better fit their problem domains, but are fully aligned with the core ideas of Lisp as defined through its historical evolution.
I also don't see let, letstar, etc as clunky but I've also been using them for more than a decade (and am firmly in the Lisp Machine Lisp -> Common Lisp camp).
So can you elaborate on what those design errors might be? And what exactly is the "concept of lisp"?
The concept of Lisp through its historic evolution refers to the set of ideas and implementation details that allows one to read a book about Lisp written 40 years ago and apply the same concepts through a language called Lisp today without any -major- changes.
If that's not possible or involves substantial programming logic rewriting, the language should not be viewed as Lisp.
Does Guix offer a branch that contains code that has been built by the CI builders? In which case, when a user grabs a package they'll be able to grab the compiled version rather than have to compile it themselves.
At this point someone should abstract out the Emacs definition so others could build it with Scheme or Clojure.
Emacs is basically a Lisp interpreter. They wrote the parts that needed to be fast in C and the rest is elisp.
45 years ago Emacs needed to be built in C. Today the entire thing could be built with a standard Lisp.
I suppose if someone wrote an Emacs in Scheme, the projects could converge once Guile is integrated into Emacs
Unfortunately that will never happen, and since packages are the reason anyone uses anything, no one will use the newmacs.
If there were a translation layer for backwards compatibility then that could work, but the one with more packages wins, so I don't think it could work without it.
I have never been limited by it. Granted, it's probably the worst lisp. But it works just fine for it's intended use. It could be argued that is one of the most used lisps.
As for porting packages, some are old and will never get ported and packages are the spice of emacs.
Of course, ES6 is less powerful than Scheme, but the speed and the mindshare outweighed the homoiconicity and the macros :-\
30 years in development do help avoid some pitfalls, though.
Someday in the far far future long after Scheme is supported and millions of lines is rewritten in Scheme, I could imagine dropping support for Elisp.
Then Emacs will be nothing more than a large Scheme program that could run on any implementation of Scheme
Most of the value in the Emacs ecosystem lies in Emacs Lisp and not in an "abstracted out" Emacs definition. Plenty of folks have implemented Emacs in various languages (using the same "abstracted out" definition) and guess what, all of them went nowhere. Because they were missing the point, just like you do.
If they were:
1 - Lispy
2 - Faster
3 - Fully compatible with existing eLisp packages.
I think people would switch.
Guile Emacs is all of those except (at this point) faster.. when I tried it, it was unfortunately way slower.
If the speed could be improved a lot and the remaining bugs ironed out, it would be a very strong contender.
Guile Emacs is not at all compatible with Emacs Lisp, it only implements a tiny subset of Emacs Lisp.
It's right there in OP's article.
So the only thing it has going for it is being .. Lispy. Not enough.
Vanilla GUI Emacs runs a non-trivial amount of eLisp code, so at least then with the version of Emacs I used (24.4.50) the compatibility situation could not have been dire than what the OP reported.
Also, you have to bear in mind that Guile Emacs was the work of one single (very talented) person over a single summer.
If there was actually a team of people working on it, I expect a lot more progress towards compatibility could be made.
Here is a list of outstanding items that still need to be worked on Guile Emacs:
The points it makes about speed are not wrong, but that the author fails to understand even the most basic relations between Guile-emacs and guile-elisp makes it hard for me to take it seriously.
AFAIK the biggest point that brought Guile-Emacs to a standstill is the internal string representation of Emacs that is different from Guile’s native format.
I think a lot of people miss the point of Guile Emacs. Making Emacs faster or having access to all of Guile's features and libraries is not what the project intends - they are just possible free perks. The purpose of Guile was to allow any project to be extensible like Emacs with an independent module. That means instead of the Emacs folk implementing a native compiler just for themselves, some GNU project that uses Lua to rewrite existing Emacs Lisp libraries, or awesome small projects like Lilypond or Gnucash to have to deal with Guile Scheme all by themselves, all these projects can pool resources and improve their performance and features together. Guile Emacs is not about Emacs, it's about bringing more unity across all the islands of GNU projects.
If I had the know how / the time I would be less talk.
I think Guile-emacs defers a lot of the core functionality to Emacs C functions as well, which would explain the failed tests.
Just a shot in the dark, but I think since so many tests are failing, all bets are off as to their performance. For example, I could see a test getting into a loop that normally only runs ten times and succeeds, but here gets to run a million times before the test aborts and gives up. Not sure that's what's at work here, but... tests typically have timeouts for a reason (and typically finish significantly sooner than their timeouts, for obvious reasons). :-)