Hacker News Reader

>On Lisp Machine, Zmacs ran in a multi-threaded Lisp

I wonder what the processing architecture of those Lisp machines were like. Did their "cpu"(?) had special circuitry for implementing lists, CAR and CDR? Was this multi-threaded-ness something built-in in hardware?

> Did their "cpu"(?) had special circuitry for implementing lists, CAR and CDR?

It was implemented in Microcode on the CPU.

> Was this multi-threaded-ness something built-in in hardware?

Some. The process scheduler was written in Lisp. See for an early example on the MIT Lisp Machine: https://github.com/mietek/mit-cadr-system-software/blob/mast...

I think you should clarify that the Lisp machines (at least the early ones that you speak of) were not 'multi-threaded' in the sense of multiprocessing--the only multiprocessing they did was running independent Lisp systems on each of their processors, if they had more than one.

From the 1981 4th edition Lisp Machine Manual, page 428ff

> 25. Processes

> The Lisp Machine supports multi-processing; several computations can be executed "concurrently" by placing each in a separate process. A process is like a processor, simulated by software. Each process has its own "program counter", its own stack of function calls and its own special-variable binding environment in which to execute its computation. (This is implemented with stack groups, see chapter 12, page 163.)

These computers could send/receive network traffic, handle the GUI, read from disk, while the user would have processes for each application and window.

On my mid 1980s Symbolics 3640 I could compile a Lisp application from scratch in a Lisp REPL for half an hour, while I was using Zmacs to edit code. The practical limitations were CPU speed, memory size, disk speed and the Garbage collector. A full GC would halt everything else. Better incremental GCs helped a lot.

In GNU Emacs last I tried I could not use two IELM repls executing Lisp code at the same time. One would block the rest of GNU Emacs Lisp code.

There is a bit to much simplification here, and confusing terminology going on here. Lisp Machines did not have "processes" (as in, Unix processes); a Lisp Machine process just a function - with some special properties.

The Lisp Machine only had one processor for the Lisp Machine, and it did absolutely support multi-processing and multi-threading.

A good explanation can be found in the Lisp Machine manual:

  https://tumbleweed.nu/r/lm-3/uv/chinual.html#Processes
  https://tumbleweed.nu/r/lm-3/uv/chinual.html#Stack-Groups

Which is also where all current work on the MIT Lisp Machine is being done. :-)

All current work ?

There is nothing special about CAR and CDR on the Lisp Machine, which should be considered a virtual machine for all intents and purposes that was implemented in microcode.

Multi processing was implemented mostly in Lisp, but stack group handling (how the system manages "threads") was done in microcode.

The hardware for the CADR is very simple, and has no knowledge of Lisp or the Lisp Machine.

> Compare with GNU Emacs, where Emacs Lisp lacks this feature. Wouldn't you want an mail thread not block the rest of Lisp code? Some Lisp action in the REPL not to block the rest of Lisp code?

Emacs Lisp does have threads; https://www.gnu.org/software/emacs/manual/html_node/elisp/Th...

Even more simply, `while-no-input` has existed since 2007. And start-process since 1985. It has always been possible to make a mail "thread".

If the developer didn't use the old tricks, it doesn't seem likely they will use the thread trick either.

ZMail on the Lisp Machine would stall the ZMail thread when fetching mail. So you wouldn't be able to compose an email if you'd be fetching 10MiB of mbox.

But it isn't so much about tricks, sometimes it is nice if software just works reliably, GNU Emacs has a great track record on that.

These "tricks" still don't make two GNU Emacs Lisp REPLs in the same GNU Emacs execute arbitrary Lisp code concurrently.

But why does that matter? Help me understand.

To be fair I haven't used an elisp REPL, unless it counts that I use commands. Any command used in Emacs is an eval step in a REPL.

But you mean a REPL as in a console-style interaction, right? It seems an impoverished interaction model, only necessary with other languages since they are not also the editor process.

Ok I can't have multiple REPLs but neither can I type on multiple keyboards at the same time. So it seems to only matter when you are going to spin up some noticeable chunk of work to be done in the background, while the user interacts with the editor. And that is totally doable with `while-no-input`.

In what situation are you running two or more noticeably slow call stacks that need to happen inside the editor process?

People seem to use Lisp like it is some impoverished programming model. Why would I want to do things concurrently in one Lisp? Because that's how I use a Lisp system all the time. That's my normal way to do things.

> But you mean a REPL as in a console-style interaction, right?

I mean > 1 REPL as an example how I would use a Lisp concurrently. Yeah, I can switch between REPLs in micro seconds and run > 1 second activities in each of them.

> Any command used in Emacs is an eval step in a REPL

And you can't run simple Lisp code in two commands concurrently, neither in a REPL nor in a command.

> `while-no-input`

Type (while-no-input (dotimes (i 1000000) (print i))) in two Emacs Lisp repls and have them running concurrently.

I understand that GNU Emacs can run external processes, external instances of GNU Emacs, external other Lisp systems, that it can provide some limited form of cooperative threads, ...

I have lived through this stuff 30 years ago and onwards, when many Lisp implementations used cooperative threading (and some had cooperative threads deeply integrated in their Lisp code, other than GNU Emacs in 2024) and over the years switched to native preemptive threading. I also remember when we got Symmetric Multiprocessing in several Lisp systems roughly 15 years ago.

That'd be better as

    (defvar foo-i 0)
    (defun foo ()
      (while-no-input
        (while (< foo-i 10000)
          (print (cl-incf foo-i))))
      (when (< foo-i 10000)
        (run-with-idle-timer 1 nil #'foo)))

Make a copy of all that, search-replacing "foo" with "bar". Run (progn (foo) (bar)). Both loops will complete eventually.

EDIT: OK, you explained what you call the normal way to do things. What kind of code is this that take >1 second in each repl? Out of curosity.

> EDIT: OK, you explained what you call the normal way to do things. What kind of code is this that take >1 second in each repl? Out of curosity.

One easy example -- listing all the files under the GNU Emacs tree to index them -- you might not want to do it many times but even `find` takes a bit of time. Looping over such a list might also take a bit of time.

> Both loops will complete eventually.

Thanks for your effort!

But I need none of this. I want to open a REPL and type (dotimes (i 1000000) (print i)) - or anything morally equivalent ;-) - and I want it to do its job and the user interface to be responsive!! There were Lisp systems, which did this 40 years ago and there are some which can do it today: Execute simple plain Lisp code concurrently without any special instrumentation. Each REPL executes by default in its own thread, concurrently. I want the same for the other tools, too.

Execute (dotimes (i 1000000) (print i)) in the Emacs Lisp REPL. The UI is dead then. I find it strange that it is dead and I find it strange that there are people who don't find that strange. ;-)

> What kind of code is this that take >1 second in each repl? Out of curosity.

All kinds of stuff.

> Out of curosity.

Have you ever used a multithreaded Lisp environment as a primary work tool?

> But why does that matter? Help me understand.

It doesn't really.

ZMacs and the Listener on the Lisp Machine is just as single threaded as GNU Emacs. One ZMacs frame cannot run the above code without locking up, one Listener instance cannot run the above code without locking up, if you open a file in ZMacs it will lock up while reading up the file.

The only difference is that when the ZMacs background process is working, you can still talk to the Lisp Machine in other ways.

The closest you might get with GNU Emacs is running multiple versions, one for each "program" (Emacs, mail, web browser, ...) -- still not exactly the same thing since you can't easily poke into the mail program from another instance.

At the end of the day .. none of this matters much, as we can evidently see since nobody has really tried to fix this.

On my Symbolics VLM installation (on a MacBook Pro) I can run things in one Listener, let it running its code and switch to the next listener. I can also use the GUI to create a new Listener. I can switch back to the busy Listener and force it into a break by control-suspend. I can also reshape a busy Listener window.

Something like the LispWorks IDE, with its editor-based listener tool, does the same.

I think Emacs' process model is generic enough that it could adopt a better method of concurrency in the future. Functions for processes are run based on 'sentinels' which fire on interrupts from foreign processes. It's a simple model which works fine, and I would greatly prefer it to be augmented with a more clever scheduler than to dragging mutexes and condition variables into the system.

> At the end of the day .. none of this matters much, as we can evidently see since nobody has really tried to fix this.

Something that's easy to take for granted is how fast Emacs actually is. Anyone who's taken the time to try out an emulated Lisp machine will quickly observe how slow and clunky it actually is most of the time. And it's worth mentioning that a lot of the crucial components of the Lisp machine system, such as the windowing system, the garbage collector and the network stack work because they are written in a more structured and low-level manner so that they run efficiently and avoid consing. It is all Lisp, sure; but there's a bit of a bias that seems to portray that its all 'plain lisp' that 'just works' concurrently because it's magic.

The VLM I'm using has the CPU emulated in assembler. Runs okay on Apple Silicon. It uses one core.

> And it's worth mentioning that a lot of the crucial components of the Lisp machine system, such as the windowing system, the garbage collector and the network stack work because they are written in a more structured and low-level manner so that they run efficiently and avoid consing.

Not all levels are low-level, there is a large higher level which is written using Flavors. Even transmitting IP packets is a Flavor method.

>sbcl

SBCL (and it's one of the fastest Common Lisps out there) ran MCCLIM at glacial speeds in my n270 netbook, kinda like a busy Pentium MMX running Windows 98 where you could see Win32 controls redrawing themselves. OTOH, GNU Emacs' multithreading sucks, but at least the UI it's miles ahead on responsiveness.

> but at least the UI it's miles ahead on responsiveness.

One Lisp REPL doing stuff and GNU Emacs is completely dead as long as the Lisp REPL runs that Emacs Lisp code. It's not even slow, it just is completely non-responsive.

This seems a bug in the design of that REPL. Unless there's some reason the process has to be the same as the editor process.

I want multiple Lisp REPLs which run Lisp code concurrently in the same Lisp process.

How would I do it in GNU Emacs?

The first question is why. Can you not use CL repls instead of elisp repls?

I would want to hack Emacs Lisp, not Common Lisp.