Compiling a Lisp to x86_64 (2020) | Hacker News Reader

Compiling a Lisp to x86_64 (2020)

bernsteinbear.com

128 pointssillysaurusx31 comments

I wasn't sure whether to submit the overview, or the first chapter: https://bernsteinbear.com/blog/compiling-a-lisp-0/

It's an exceptionally high-quality series, made by HN user tekknolagi: https://news.ycombinator.com/item?id=26627784

Aw thanks ^_^ Happy to answer questions and receive feedback

what happened between 'writing a lisp' and 'compiling a lisp' that made you switch to C from OCaml?

I use C/C++ for my day job these days, instead of OCaml. That's probably the primary reason. But also I wanted to remove a couple layers of abstraction.

Is 2021 the year of Lisp on the desktop?

Depends on how hard you want to stretch the definition of "Lisp" here. But I guess if someone would make a proper Lisp for CLR and marry Emacs with PowerShell, then we'd be half-way towards resurrecting the corpse of Lisp Machine into a functioning zombie.

That's quite a... disturbing... picture you're painting... :D

The dream.

Clojure CLR is a thing. And despite the opposite opinions, Clojure is very much "a proper Lisp".

I'm still confused about the state of Clojure CLR. I initially thought it was the "OG Clojure" before Rich dumped CLR and went for JVM instead. Is it maintained properly and kept up to date with Clojure proper?

(I used to be in the "Clojure is not Lisp" camp, but I've grown up at some point.)

My understanding is that the Clojure community points everyone to Arcadia[0] since it's maintained and a bit more public about what their exact goals are. Unfortunately, neither are terribly well documented and so I've not personally used either

[0]: https://github.com/arcadia-unity/arcadia

> I'm still confused about the state of Clojure CLR.

David Miller been doing a tremendous job of keeping it up to date. Latest commit was 2 days ago.

Somehow there's not much overlapping interest there - dotnet folks aren't very much interested in Lisps. Lispers don't care much about dotnet.

And VS Code with Calva make REPL-driven Clojure development a very pleasant experience.

Edit: With the JVM version, I've never tried that setup with Clojure CLR.

I just tried it out of curiosity and the startup time is absolutely brutal:

    -▶ time dotnet ./Clojure.Main.dll <<<""
    Clojure 1.10.0-beta1
    user=> user=>
    dotnet ./Clojure.Main.dll <<< ""  16.74s user 0.18s system 99% cpu 16.980 total

Compared to Clojure on JVM:

    -▶ time clojure -e '(System/exit 0)'
    clojure -e '(System/exit 0)'  3.02s user 0.16s system 268% cpu 1.184 total

I wonder, did I do something wrong (I'm not a .NET developer by any stretch) or does it simply take that much time on the CLR?

.NET may suffer from the same problem as Julia: the first runs will trigger the jit system and result in awful start times. There are ways to avoid this, but I don't know if it works (yet) for clojure.

I have this somewhat in linux.

I use Guix System, which lets me use scheme as my configuration, services, and package management language.

I use stumpwm as my window manager

Emacs as my "shell", text editor, word processor, email client, and news reader.

Nyxt as my web browser about 70% of the time.

Its very much a frankenstein, considering I'm using three separate lisps to configure and play with it, but it makes me happy.

Oh, that's nice!

How is Nyxt? I've been meaning to try it out. Is it usable for day-to-day tasks? What are the remaining 30% of your use?

As for Guix, I've approached it twice, both times installing (what was then known as) GuixSD. Couldn't really make it work. What are your experiences? And your configuration? :).

Emacs, that I swear by.

Still, what's missing in this picture, and why I mentioned PowerShell, is a typed integration with entire OS. Operating on objects / typed data, invoking functions - instead of passing unstructured blobs of text around. That's what I like about PowerShell approach - my scripts don't invoke CLI utilities, they ask system for objects, inspect them, call into their methods. This is similar to the Lisp Machines, where you could code against objects representing your OS runtime state.

If we could somehow bolt that on top - replace existing Unix tools with ones accepting and producing structured data (or even wrap them) - then the picture would be complete. And I wouldn't mind using four different Lisps, if that's what it takes :).

The last 30% of internet usage is work related stuff (checking email, issues for our internal git, time entry, etc. stuff that’s only available to “approved” browsers).

To make guix system work I had to compile my own proprietary kernel and build an installation image for it. Afterwards I’ve only had minor software and driver issues. I even have steam running now.

So I hear you and through stumpwm I guess I get to work with structured data for window management. It isn’t typed but windows are represented as objects to be manipulated. Thanks to the repl I can change window behavior on the fly and have it reflected immediately.

Guix provides structured ways of manipulating system services and configurations. Most of that stuff has an interface with scheme records.

For instance my sddm configuration is a scheme record that gets tangled (for lack of a better word) into a text file.

With guix one can also use g-expressions to write configurations or execute programs in a monad of some sort.

It isn’t typed but the setup is far from unstructured.

For playing with a Common Lisp desktop, try Mezzanine on x86 hardware or a suitable VirtualBox.

https://news.ycombinator.com/item?id=8943709

Thanks for the first lesson, I'm learning C after avoiding it for 4 decades (because of Cosmopolitan/APE) and I really needed to learn how to make self modifying code in C.

Your explanation helped out greatly.

This is one hell of a way to learn C but I'm glad you're enjoying :P

First, I just have to say what an amazing resource this is. Thanks!

But I do have two questions, the first of which has literally burned a hole in my brain since I read your tutorial: When you emit the instructions to the buffer, I noticed that you encode the objects(or at least the ints) beforehand. Why is this? I am specifically talking about about line 3 in compile_expr() in part 2. Am I misunderstanding something?

My second question is when you might be releasing a new part in the series ;)

1) Great question! This is because there is no type information in assembly. We have to build it ourselves. There are sort of two ways to do that: one is to have the compiler know everything ahead of time about what types registers/stack locations/etc all have. The other (which I chose to use) is dynamic typing, where the type information is somehow encoded into the object. In this case, the type information is encoded into the pointer. So when we generate instructions like "mov rax, <something>", we have to encode that the object is an integer.

2) Yeah, I know :( I had some life events in fall/winter that stopped all my motion on this series. I have a half-working closure implementation -- the actual closures are fine but the tree transformations are not yet implemented.

Ah, I see; I am not an expert in this field so it isn't really clear to me why that information would be needed; the assembly is already being emitted? Is it because functions that will take ints(which will inevitably recieve them from some kind of stack pop or similar) have to have the typing verified before their execution begins?

I hope your situation has improved/is improving; good luck! I love your website.

Well Lisp can reflect on its objects! You can have code like:

(int? foo)

that returns #t or #f depending on if foo is an int. So `int?` needs to have information from somewhere about what type of object it is.

I get more into this in Part 4 (unary primitives).

Thanks :)

Nice! I recognize this domain name - you were working on a stack language I think(?) many years ago. Cool to see you're still at it.

That's me! Probably.

Carp seems to be a modern compiled lisp with some traction.

With an interesting ownership model, no less.

Can it be arm as well?

And it would be better if both are c?

I was looking into doing ARM or RISC-V but didn't want to get distracted from finishing the series with one architecture first. But here I am, series unfinished anyway...

And I assume you are talking about writing vs compiling a Lisp series? They were written at two different points in my life, and they implement two different languages at this point.

Reply on news.ycombinator.com