Comment by DDSDev | Hacker News Reader

Comment by DDSDev | Hacker News Reader

Parent Full thread

In response to the general critiques of "Why rewrite emacs at all!?", I believe there are hidden benefits to an incremental rewrite to Rust executed this way.

First is that remacs has been able to reduce the amount of non-Elisp code in the editor needed to implement a number of different features. It was able to do this by replacing hand rolled C functionality with high quality crates from cargo, or code in the standard library. I'd attribute this to how easy the cargo system makes implementing libraries into your project.

Second is I believe that this project overall is increasing knowledge share of not just Rust, but also mainline emacs. I personally was intimidated by looking into contributing to mainline emacs. The remacs community was so friendly and welcoming I felt very encouraged to jump in and start working on things, and now I not only have better knowledge of Rust, but a pretty decent knowledge of the C layer in emacs and how it lays out and implements it's data structures etc.

Third is that the majority of the emacs codebase is written in emacs Lisp. Remacs aims to be fully Elisp compatible, and any differences are considered bugs and should be reported. This means that the project is still able to keep up feature and bug parity with the mainline, while incrementally rewriting the core of the editor.

Disclosure: I am a contributor to this project

There is a huge amount of C code in Emacs that already isn't using the standard C library. As an example, the other day I noticed that Emacs has its own hand rolled implementation of mkstemp(). Unsurprisingly, the Emacs version uses a less secure implementation, so the custom implementation is of no value.

A lot of people have already written about the difficulty of getting changes to fix things like this (or eliminate dead code, etc.) landed in upstream GNU Emacs. Rewriting parts of Emacs in Rust is an interesting idea, and I applaud it as a fun project. But for people who are interested in making more incremental improvements to Emacs, there's a lot of work to be done in the existing C code base to make it smaller and more secure.

I don't see how it is any more incremental than what Remacs is trying to do. They're doing a parasitic rewrite of GNU Emacs, it's not a from-scratch implementation.

If it's hard to upstream code in emacs, doesn't that improve the case for a rewrite/fork?

The mkstemp you are seeing is probably the one from gnulib. A lot of GNU software uses their own versions if they detect the platform one is broken; it may be incorrectly replacing it in this case.

https://www.gnu.org/software/gnulib/manual/html_node/mkstemp...

https://github.com/emacs-mirror/emacs/blob/65eee8392ff95f58f...

To be fair: emacs was written in an era where the idea of a "standard" C library was kind of a joke. Those functions provide "no value" today on systems that have been POSIX-valid for decades. They certainly did in the mid-80's when Unix portability was the biggest hassle of the day.

That said, though: wouldn't that argue for things like "remove custom mkstemp" and not "rewrite it all in rust?"

As more of an admin kind of guy, i find this explosion of language specific repositories worrying.

I have already had to deal with python code that do little to no dependency documentation, that just carpet bomb my system with stuff pulled from the net at the install stage.

(I hope I'm not misunderstanding your concern), but a quick search reveals that Rust prefers to statically link, so in this case, remacs would distribute as a 'single' binary. Only the developers (of remacs) would need all the dependencies/crates.

This creates its own headaches as a fix for the next Heartbleed becomes not "upgrade this one shared library", but "upgrade these dozens of individual programs when an update becomes available".

There has always been war b/t static vs dynamic linking. The debate will continue forever, but it doesn't really matter anymore. Each side understands and has solutions to its deficiencies.

Fwiw, lisps have a somewhat interesting third way: you distribute everything in a single image like a statically linked program. But, since that image includes the compiler, it can upgrade itself.

Or they include a loader and can load code, code which may be compiled somewhere else.

Programs all need to update all the time anyway.

with python the rule should to be to run every deployment at least in its own venv.

That's not so practical if the application is a text editor, or some other administration tool (debugging, statistics etc).

My rule of thumb is to never install anything into system Python by pip, etc. If it's not an OS package, it belongs to a virtualenv.

I more or less have this rule, except I generally will use the --user flag to install things on a per-user basis if it's not being installed for a particular project.

I recently came across this: https://github.com/spotify/dh-virtualenv

For a system utility, you can still use virtualenvs. The launcher script just needs to use a shebang like:

    #!/usr/share/spiffyapp/venv/bin/python

Python has/had a terrible packaging story though. As you said, dependencies weren't really documented. Rust, on the other hand, documents all dependencies in Cargo.toml. And dependencies can easily be parsed from Cargo.toml without the need to execute any code (as would be required in the case of python)

> Python has/had a terrible packaging story though. As you said, dependencies weren't really documented.

I wonder where have you lived? I rarely found (if ever) an undocumented dependency list for any serious Python project in the last couple of years.

Are you talking about requirements.txt? Requirements.txt falls short of cargo or really any other mainstream language's dependency doc (no reproducible builds, for one thing -- effectively almost as bad as undocumented).

Requirements.txt + virtualenv, everything is fine.

Really, significantly easier to not screw up in a large org than bundler, because dependencies don't leak by default.

Actually, `bundle exec` is way easier to use than virtualenv.

Mostly requirements.txt, yes. It's a standard and it has been a really long time I used a package where I didn't find one.

What you mean by no reproducible builds?

python's packaging, especially compared to state of the art which cargo represents, is really, really poor. documenting direct dependencies is just a start. requirements.txt is somehow simplistic and complex at the same time (remember about the special options which thankfully aren't used too much) and you can also list dependencies in setup.py, too.

there's the Pipfile initiative which gives me much hope, though :)

It really does surprise me that languages come with their own package managers. In most linux contexts, there's really no need for it since linux distributions come with package managers.

Part of me feels like a better solution is to maintain older branches to give distributions opportunity to catch up.

> In most linux contexts, there's really no need for it since linux distributions come with package managers

This single sentences hints at least major two reasons for language-specific package managers:

- non-Linux platforms

- there's many different Linux package managers (apt, rpm, pacman, nix, portage, ...)

A language-specific package manager means developers (i.e. the ones writing the packages) only have do to packaging work once, instead of for every platform they might want to support, including ones that they have no way to test on (and a lot of code, especially in the modern wave of languages like Rust, often works on all platforms with no particular effort from the developer).

From an asymptotic perspective, using the platform package managers results in something like O(# libraries * # platforms) work and likely a lot of code not being packaged for some platforms, whereas language-specific package managers is something closer to O(# languages + # libraries) work (or maybe O(# languages * # platforms + # libraries). Looking at this decision purely in terms of developer effort, the latter seems like the better choice.

(There's also things like fpm that are designed to be more like O(# platforms + # libraries) work, which is nicer than either of the above!)

For anything but C, one can't really trust the distributions to care enough for that to be an option. You can install Common Lisp libraries from the Debian repositories, but if you do, you'll find yourself in a world of problems since they are mostly severely out of date, if they work at all.

Far too many times have I had to explain to newcomers to Common Lisp that they can't install stuff from the repositories, and instead have to download their CL implementation separately and then use Quicklisp.

language specific repositories worrying

What is the alternative? Every major Linux distro has its own repository, as do all the BSDs. The picture on Windows and macOS is even more convoluted. Are developers in all these languages supposed to support the packaging of their software in half a dozen different repositories?

Maybe all programmers should get together and build "one repository to rule them all"? I have no idea how we herd that many cats!

Sounds like xkcd 927 ...

FPM is at least a step in the right direction to reduce the overhead of creating all of these.

At a minimum, Linux should have a common packaging system; though there may never be agreement. It's hard to see macOS and windows endingnup with a common one.

But maybe we could do a third party option that is independent of each OS? Those rarely work out well though.

And that is forgetting that are many more OSes out there than just those three.

Of course there are many more, but if you could build the one true package manager for all the major OSes, my guess is that the smaller ones would join the fray.

Not really, because it would meant that their semantics weren't accounted for.

As an example, IBM just released yesterday a new mainframe model.

z/OS is a totally different animal compared with POSIX based OSes or Windows, starting by the file system, catalog based.

Nix(OS) is basically that... If only we could get upstream to start building with Nix

Ugh...no. I tried Nix until I found that all packages that depended on OpenSSL were broken on OS X. I checked for a bug report and found that one had been filed and sat open and unfixed for 18 months. Given that this bug makes Nix basically unusable on OS X, it seems like developers of Nix are uninterested in becoming anything resembling a universal package manager.

Meanwhile, cargo works flawlessly on every system I've tried (Linux, Mac and Windows). Sure, it only compiles Rust programs, but there's no way that I'd advocate moving those programs to a package manager with as little regard for such a large part of its potential userbase.

> Given that this bug makes Nix basically unusable on OS X, it seems like developers of Nix are uninterested in becoming anything resembling a universal package manager.

Or that they just don't have enough manpower to actually support all the platforms.

This is a really great description of the benefits of this sort of thing. In a code base like Emacs, I could easily see where learning the guts could be very hard / daunting for a new-comer (I've never peeked under the hood for exactly this reason).

You all are scratching an itch, and increasing the general mindshare for Emacs. If Remacs runs out of steam, then the world has some extra people who grok Emacs guts. If Remacs fully succeeds in remaking Emacs, but better, then maybe Remacs just becomes Emacs.

> Disclosure: I am a contributor to this project

Thanks for posting this and thanks for your contributions.

Reply on news.ycombinator.com