Racket Compiler and Runtime Status
blog.racket-lang.org
blog.racket-lang.org
Can we appreciate the meta? A fast low-level program was re-written in a high-level language to reduce maintenance costs. These days on HN you usually only hear about the opposite, eg. Figma rewriting server Typescript code in Rust to improve performance [1]. Not fair to compare Rust to C, I know, but interesting nonetheless.
The loss of JIT was interesting. How does eval work? Apparently Chez Scheme AOT compiles code at runtime. Comparatively a JIT will change the compilation at runtime based on codepaths. There's a thread about that here [2].
Part of the motivation of CS was to make it easier for people to contribute to the compiler. I'm excited to see how that plays out.
Thanks again everybody. I'll go back to toying around with macros and tiny Racket programs now.
[1]: https://www.figma.com/blog/rust-in-production-at-figma/
Related, though not quit the same, is the manner in which the Cog VMs for Squeak/Pharo are developed:
"The VMs are developed in Smalltalk, using all the dynamic and reflective facilities of the Squeak/Pharo Smalltalk system. As such, developing in Cog is a delight. The Smalltalk framework comprising the various Cog VMs is translated into C by its Slang component to produce VM source that is combined with platform-specific support sources and compiled via a C compiler to obtain a fast production VM."
And as a side effect make it faster on average. Very cool!
https://github.com/cisco/ChezScheme/issues/545
Racket-Chez is different from Cisco-Chez by > 600 commits. I wonder what the plan is to normalize them or will be Racket-Chez be our new house.
I don't think it's likely that we'll move back to pure upstream Chez in the near future, but it's not quite a fork either.
You can read more about some of these issues in our ICFP 2019 paper: https://www.cs.utah.edu/plt/rkt-on-chez/ which is co-authored with the Chez maintainers.
I'll read the Racket on Chez papers. Are there plans for more backends? RISC-V, Wasm or Scheme (not Chez specific)? Wasm would be interesting as that would also enable a path to integrate with Graal. I understand that Wasm doesn't yet support native tail calls, so that would have to have to be addressed.
> A Racket-implemented layer of Racket CS must be translated to Scheme to run on top of Chez Scheme.
That contract between Racket and Chez for how Racket runs on Chez, I think that is fairly interesting from a language engineering perspective. It seems like it could be a nice RRRS (Revised Report on Racket on Scheme).
What are your thoughts on Shen?
Wasm would be very nice to have; I think compile-to-web is a big missing piece although there are potentially other ways to do it.
For Graal I think a direct implementation would be nice.
Compiling to other Schemes would be great, and I think we're in a good position to make that happen now.
I think the contract isn't that interesting in a sense -- it's just most of Chez. But the paper does talk about that sort of thing.
GNU Guile? Common Lisp? Clojure (JVM)? Racket? Judging by my understanding of the difference between Common Lisp and Scheme, I think I am more of a Scheme type (I prefer C over C++, I like Go more than Java, etc).
Clojure if you're looking to use the language directly to solve your problems (and my preferred Lisp).
If you care about startup times or FFI then don't pick Clojure.
YMMV between Racket and Guile; I would say Racket is better insofar as the culture of documentation is really very good.
In practice, once you learn a Lisp, you can jump between them without too much trouble.
Isn't Clojure in some significant aspects similar to Schemes?
I use guile since I think it more fun, but racket is a lot easier if you want libraries around.
I managed to patch guile to fold equal? to eq? when the comparisons involved literal symbols, chars or fixnums. (Although maybe not in the best way, it still worked). I apparentlynnerd-sniped Andy so he implemented it properly as a part of the expander (I think). That took me less than an hour starting from a cursory understanding of the guile codebase.
In terms of schemes you have a wide variety to pick from. Gerbil, Racket, Gambit, Chez, Chicken and many more. Racket is a great all around choice. There are schemes that do various things better than it, but it can do most things well. It has two great edX courses to learn from (How to design simple data, and how to design complex data) and is how I initially learned.
Chicken is one of my favorite schemes and is probably the one I would use if I wasn't using Racket. It's so portable since it compiles to C. I love the egg system, and I like the logo. Gerbil is a performant systems level scheme. I remember there being an article about someone in the Common Lisp community considering jumping to Racket, but ending up on Gerbil Scheme for what it's worth. I haven't used it
Chez is another super performant scheme, and is actually the backend for the Racket programming language as seen in the article above. It was only open sourced a few years ago so it might be difficult to find solutions or get answers. I've used Guile, I appreciate the mission, but I found it wanting. I've heard people ship Gambit scheme apps onto IOS so if you are looking at mobile apps that might be the way to go, I'm sure Chicken could do that to. I have also heard that Gambits Cffi is one of the best in the business, so if you need to interface with a lot of C code that might be the way to go.
https://mitpress.mit.edu/sites/default/files/sicp/index.html
After getting comfortable with Lisp based languages, it is like Algol derived ones, it is relatively easy to jump between them.
I tend to stay with Clojure, because JVM/CLR is where I spend most of my time, so I can easily pig back into the libraries.
Maybe what you can do is similarly, pick a Lisp variant for the domain you spend most of your time on.
Scheme is not more minimalist than C.L. in practice; it's standardization is simply more fragmented and has nested layers, often by different organizations, that implementations can support, and then most implementations go well above even that.
Different Scheme implementations tend to share a common base but then tend to provide functionality above that, often shared between different implementations as well.
So to explain just a bit, often I do a lot of paralysis by analysis of tooling, but as a side effect one of the things I've learned to help me get past it is to filter first based on licensing. I like Nixos, and I think the principles there are going to go far into the future, but I don't like MIT/BSD licenses. So if I want the same kind of tech, but in GNU-land, that leaves me with GUIX (the os and the package manager). Decision made.
Only in the most extreme circumstances do I violate these principles (steam for gaming!) and am always looking for alternatives as they pop up.
I really like the idea of compiled lisp eg common lisp, but most of the gpl friendly versions are not in active development, besides maybe Clozure CL. Clisp website shows last update was in 2010, and for some reason that really rubs me the wrong way.
It's funny because I still really don't know that much about lisp, it's just on my "to get into" list, besides elisp.
https://fedoraproject.org/wiki/Changes/RemoveGuileFromToolch...
Apparently they dont like it that much
Here some axes of distinction:
1. supported programming styles
2. support for concurrency
3. performance of generated code and parallelism
4. floating-point performance
5. level of standardization
6. closeness to the system and capability to call into C functions, or functions with C ABI
7. suitability for scripting and stand-alone programs
8. GUI programming
9. Comprehensiveness and beginner-friendliness of documentation
10. REPL Programming
11. Libraries
12. Licenses
Here what I know about the languages you name:
(I also mention ABCL = Armed Bear Common Lisp a few times, just to illustrate that you also can run Common Lisp on the Java Platform).
1. supported paradigms - Clojure is very opiniated in supporting and demanding a purely functional style. It uses purely functional data structures. Racket and Scheme support a functional style well, but allow for seamless imperative code. Common Lisp is agnostic, one can program in a purely functional way and there are libraries with purely functional data structures, but it requires much more discipline.
2. Clojure has best support for concurrency and server-like tasks, it is made for that. Racket also supports green threads, apart from its places parallelism.
3. Racket, Chez, Common Lisp and so on support OS level threads and parallelism. Chez and Common Lisp stand out as they generate the most performant code - when it comes to raw computing power, single-threaded SBCL code will be faster than multi-threaded Clojure code. SBCL, for example, also allows to add compiler hints which generate unsafe code with better optimizations (for example, indicating that integer values will only be in a specific range).
4. SBCL has by far the best floating-point performance, I think Chez comes after that, and I'd expect Racket to improve further here. Racket has also been reviewed very favourably for scientific applications (https://khinsen.wordpress.com/2014/05/10/exploring-racket/ - Konrad Hinsen, the author, was an early contributor to Numerical Python).
5. Of all these, Common Lisp is standardized most. Especially Racket and Clojure are defined by their implementation.
6. Clojure and ABCL allow to call into Java. In turn, all of Common Lisp, Guile, Racket, Chez allow to call easily into C. There is a performance difference, I think, between most Schemes and SBCL: Calling into a C function has some extra cost because the stack is handled differently (I think it is because of continuations). I also tried to call into Rust functions from Racket and that works very nicely.
7. Clojure's startup time is simply too slow for scripting. It is also a bit hampered in that it runs on the JVM. There is babashka, which is interesting but has no mature status. Conversely, Common Lisp and Racket are well suited to scripting. It is also possible to compile SBCL and Racket programs into a single executable. I think SBCL supports this case best, for running Racket programs, an extra runtime library or a standard Racket installation is needed. Guile is also well-suited to scripting and is closer to the OS than some other Lisps.
8. Racket has a very nice support for GUI programming with a platform-independent functional API. Clojure and ABCL also allow to call into Swing and Fx code written in Java. I have not tried the rest but it looks relatively painful and brittle in comparison.
9. Racket has very comprehensive and high-quality documentation good for beginners. Clojure is also comprehensively documented, but might assume a bit more experience. Common Lisp is, as an open system, more eclectic, and this might lead especially beginners to underestimate how mature and good it is. It is very under-hyped compared to Clojure. But there is the Common Lisp Cookbook on the web, which is really good. I can also warmly recommend the books "Practical Common Lisp" by Peter Seibel, and "Common Lisp Recipes" by (corrected!) Edmund Weitz. If you want to have true understanding, and are about to write industry-grade robust software, they are really worth every cent.
10. REPL Programming: Lisps and Schemes such as Racket have a subtle difference which affects the style of programming. In Lisp, everything is dynamic and you can re-load and evaluate parts of the program when it is running. This makes for a great development environment. In contrast to that, Racket for example has a clear distinction between compilation time and run time, which gives certain safety and correctness guarantees, but requires that the program is re-loaded more often during development.
11. Libraries: Racket has a distinct "Batteries included" feeling which makes it nice for beginners. Clojure uses Leiningen which runs on top of Maven to automatically retrieve libraries, it works very well. Racket has a similar system. Common Lisp is more eclectic again, what is used today is quicklisp and asdf (see the Common Lisp Cookbook). There is also a reasonable support in OS Packages e.g. in Debian, and in addition GNU Guix is a very interesting solution for packaging of Lisp libraries and in polyglot projects.
12. Licenses: Clojure has a non-copyleft license (Eclipse License) which might be an obstacle if you want to publish copyleft libraries and code (not sure about the exact limits here but the problem is that in a Lisp system, there is less clear of a boundary of system or language code and your own code, so it is probably advisable to use a language with compatible license if you want to publish copyleft code). Racket is Apache 2 which is GPL compatible. Guile is, as part of the GNU system, GNU LGPL.
In summary, I think Racket is great for beginners, although you could start with all of them. Clojure is fantastic for servers and to learn about functional programming. Guile and Racket are great for scripting. Common Lisp, and especially SBCL requires a bit more learning effort at the beginning, but it is a really mature and highly performant system which gives you a lot of freedom.
Guile and S7 are great for embedding in C programs. S7 is similar to Guile but permissively licensed.
Gambit and Chicken compile down to C if you want to work that way. Gambit is very fast, Chicken is extremely well documented and beginner friendly.
Clojure has by far the best books out for enterprise development and web dev if you want to build "normal" business apps and like books.
In the same way, Common Lisp uses Quicklisp, which runs on top of ASDF to automatically retrieve and compile libraries. It works very well.
One difference is that Quicklisp + ASDF often needs to be set up manually which can be a little bit more complicated (it is, however, contained in Debian as a package, for example).
Leiningen and Maven relay on version numbers for dependencies.
My understanding is that Quicklisp in practice uses a "rolling release" model similar to Arch Linux - you get a current distribution, where all packages are at their newest version, and if needed, one can also roll back to earlier versions of the whole distribution. That this works with elegance is related to one aspect of Common Lisp culture which is very, very different from, say, Python or Go: Common Lisp puts a lot of importance on backward compatibility and stable APIs, it is possibly the most backward-compatible of all dynamic languages.
Further, while Leiningen is a de facto standard which makes working with Clojure libraries essentially frictionles, Quicklist is just one modern way of multiple ways to include Common Lisp libraries.
This difference is owed to the trait that Common Lisp is much more an open system (which I consider a good thing), while Clojure is more uniform and "managed".
Another thing which may be nice to talk about is where the communities are for each language (Racket on Slack and the mailing lists, Common Lisp has an irc...).
It is clearly geared to servers, for example sequences are always lazy. This is not always the best thing but for servers, it seems in general better.
However, there is a subtle point about parallelism: One typically wants concurrency and parallelism for one of three objectives:
1. Concurrency in servers
2. Graphical user interfaces
3. exploiting many cores for complex parallel computations, like number crunching
It turns out that Clojure is great for the first application. It also works for the second one.
However it is not that great for the third area of interest. The reason is performance. For example, floating point numbers passed in function arguments are often boxed. Also, due to the JVM it is not possible to pass even small objects by value. These things put a lot of pressure on the garbage collector, even if the Clojure compiler partially tries to work around this.
And the result of that is that a single-threaded program which does things like massive number crunching written in Common Lisp will be faster than a parallel program written in Clojure.
But I still think that immutability in-the-large is the right way to go.
The real Racket language is a general purpose language with all the bells and whistles you could ever imagine.
DSLs are just better integrated into the base language so your source code doesn't look like:
three = two.add(one)> It would be nice if porting to Chez Scheme made every aspect of Racket magically faster. It hasn’t done that, but we have plenty of room for improvement; the performance results to date are a lower bound on performance, not an upper bound.
> Keep in mind that the original goal is not to have a faster Racket, but a better-implemented Racket with acceptable performance. The Racket-on-Chez implementation is far more maintainable and flexible than the current Racket implementation, so maybe we’re half-way there after one year of work.
The fact that the performance looks this good is a huge win. And the focus on maintainability over speed is a refreshing sight. This is the sort of practical solution that will allow Racket to be relevant for many years to come. Racket is a good all around Scheme, there are other schemes that do certain aspects of what Racket can do better, but as a whole Racket is a very complete package.
I have two main criticisms with Racket, which are totally fixable, and in large part due to Racket's history as an incubator for language oriented programming, developed by PhDs in Comp Sci. Racket is very Academic. The documentation is great, but it is written by other Academics and can be hard to grok. Same with example code. Sure if you are looking at the student languages and tutorials based on that it is more accessible, but the minute you are trying to write a complex GUI app in typed Racket, the tutorials and example code is few and far between. Sometimes it makes me feel like I'm to dumb to be using the language, and I feel like I'm not the only one to get that feeling.
Also for a language that prides itself on Language Oriented Programming, I want to some more examples/tutorials that are not just using brain fuck language. These are fixable, and I admit some are just me problems, but I'm actively working to solve them in this space.
I'm excited about Chez for the performance gains, and I think it's really nifty that it allows more of Racket to be written in Scheme.
In scheme, despite not being traditionally multithreaded as a standard, any computation can always be returned to several times due to call/cc. This means that even many cases of hidden otherwise "safe" mutability might lead to bad results.
Which is why scheme tends towards immutability. That is the safe thing, culturally and also by design. Whereas the happy path for python is generally one of mutability, the happy path for scheme is not.
Which is also something that I try to keep in mind when learning a new language. It is very tempting to try to write scheme in another language, but doing so in python will lead to sadness.
I have seen so many people trying to learn scheme by trying to write python in scheme, which usually ends up being awkward, slow and buggy in addition to not working with call/cc or delimited continuations.
I had an epiphany about this when trying to write scheme in rust. I took a step back and realized that I was the python programmer :)
Clojure with its focus on immutability and its purely functional data structures makes it much easier to write correct parallel and concurrent programs.
However, there is a subtle point: One typically wants concurrency and parallelism for one of three objectives:
1. Concurrency in servers
2. Graphical user interfaces
3. exploiting multiple cores for complex parallel computations, like number crunching
It turns out that Clojure is great for the first application, and that it works very nicely for the second one (once you bind a GUI written in, say, Swing, to Clojures promises and actors).
However it is not that great for the third area of interest. The reason is performance. For example, sequences in Clojure are always lazy, and floating point numbers passed in function arguments are often boxed. Also, it is not possible to pass even small objects by value. These things put a lot of pressure on the garbage collector.
And the result of that is that a single-threaded program which does things like massive number crunching written in Common Lisp will be faster than a parallel program written in Clojure.
This is a drawback which is rooted by the implementation.
But I still think that immutability in-the-large is the right way to go. Common Lisp libraries and Schemes are already adopting purely functional data structures, and I think this principle will become more important in the future and their possible offspring.
It's kind of an oddball. While the development focus is mostly educational, specifically Programming Language theory, it's also very usable in the real world. There's a few tutorials on writing your own domain specific language in it. You can specify a language other than the default Racket, so one compiler/interpreter environment actually supports many Lisps and DSLs.
I'm a long time dev who recently picked up SICP to learn why everyone says learning Lisp/Scheme will make you a better programmer, and Racket is by far the most interesting environment I've worked in. I haven't even touched Python, which was my money-maker, in months.
You know what they say about emacs. "Emacs a great operating system, it's just lacking a decent text editor." Same with Racket. Racket is a great language development framework, the only thing missing is a decent language to write your program in.
It does take a lot more effort and expertise to design a language and a program. That's why it's fun for me, I get to dream more and need to learn more.
I'm inspired by the super interesting Racket papers [1] and PG's book "On Lisp". It helps that I got to use Racket to make a couple basic compilers in some uwaterloo classes too.. but I didn't even use macros for those projects, so I haven't really gotten started with Racket yet ;)
[1]: https://www2.ccs.neu.edu/racket/pubs/scheme2007-ctf.pdf
It also has a Big Idea; it aims to be the practical realization of the grand idea that Lisps are the perfect platform to build programming languages on, and to take this so radically far that you can have entirely different programming languages that nevertheless share, not just a runtime, but all their libraries. Unfortunately, so far, nobody except Racket enthusiasts appear to have much interest in building languages on top of Racket - but it's absolutely possible and practical, and someone really did accomplish the astonishing feat of implementing Python (Python 2, alas) in Racket, in such a way that you could combine libraries from both languages.
Congratulations to everyone involved into making this happen.
Check 'Getting started' above, but I'd suggest that if you are already a developer, start with the Racket Guide, and 'How to Program Racket: a Style Guide' both available in the documentation site https://docs.racket-lang.org (and bundled with the installer). The Racket Guide is fully integrated with the Reference - click on something and it will take you to the reference for the full definition.
It's also worth working through HTDP https://htdp.org - with the caveat that you are using the teaching languages. A really nice thing is Racket can support students/learners, without compromising the full language. The Racket installer includes Racket, proper, the teaching languages, and a number of other languages.
https://www.google.com/url?sa=t&source=web&rct=j&url=https:/...