Why Turtl switched from Common Lisp to JavaScript (2019) | Hacker News Reader

Why Turtl switched from Common Lisp to JavaScript (2019)

lisp-journey.gitlab.io

85 pointse3bc54b269 comments

I’m interested in other people’s thoughts on this.

In the Lisp world we often talk about “exploratory programming.” There are contexts, or at least there have been in my career, when I sit down to work on a particular problem and I don’t know ahead of time how it’s gonna work out or how I’m going to solve it. In these contexts I find Common Lisp to be a useful tool.

In other contexts I have a clear understanding of what I need to do, and so writing software in C++ or some other language is fine.

I’m not really interested in trying to shoehorn one solution into another context. I’d say use the tool that works best for you in the context in which you find yourself.

I could be wrong. Interested in your thoughts if you want to share them.

Exploratory programming, for me, involves tons of quick refactoring jobs. Moving methods around, copying code from one place to another. If I sit down with a decent Go, C#, or Java IDE, I can sling code left and right, mash it up, refactor it, move code around, change interfaces, etc.

The good tooling for these languages makes it so I can do a lot of this work with a couple button presses. I inevitably end up with code made from different phases of the exploration... the static type checker is good at reminding me which pieces of the code base need to be updated, because they were from an earlier phase of exploration. There are a lot of idiosyncratic tricks buried in here, like changing the name of a method when I change its semantics (like frobnicateWidgets() to frobnicateWidgets2()), so all the call sites become errors, and I can review them one by one.

This works poorly in C++, the tooling just sucks.

I agree that tooling for JVM languages, Go, and C# are better at that, but clang and Jetbrains' CLion has helped close that gap. They've done a lot to make refactoring a lot easier, find errors better, and not have to compile just to find obvious mistakes. Newer languages are much better designed for IDE/tooling support in general, with stronger type systems in a lot of cases.

Jetbrains has put a similar amount of work into tools like Rider, it's just that they're starting from a language where these problems are much simpler to deal with.

I do it all the time using C++. The VS Code tooling is good enough for me.

I do exploratory programming all the time in C++ and Haskell. I know Lisp (scheme, Clojure) however I prefer typed languages. Types enable me to sketch out a whole application without writing any code. The types will tell me if the architecture is fully specked. And when done, the implementation step is simply to add the code needed to implement the type contracts. I often implement code with zero or very few bugs that way.

While I agree and do the same, I guess when something you integrate with is untyped (hardware, bags of (crappy) json or xml etc), exploring really helps.

Strange, there is a great web server woo [1] and great parallel futures library for async lparalel [2] that we use with great results.

[1] https://github.com/fukamachi/woo (4x faster than nodejs) [2] https://lparallel.org/

As far as I know, Woo has no async foundation, meaning it can serve static content very quickly, but cannot really make use of other async libs without farming out to another event loop/thread pool.

In other words, its main asset, speed, is bottlenecked as soon as you need to do things.

Wookie, the original server for Turtl, was built on top of cl-async which gave it access to all the other async libs built for CL. Woo wouldn't have been particularly useful for us.

That would be a misunderstanding of Woo. You can have as many workers running as you like with woo, but for async activities processing requests, the issue isn't woo, that is the server doing things with the request once received, and for that you run lparallel with great results. You can simply send every request off in a future, and it will go off and do it on another thread.

So you're not running anything truly async, you're just using threading at a lower level? I don't get what Woo really provides to the async ecosystem in CL.

Woo is a superfast web server. It handles receiving 4x the number of requests that node js can. Once the server gets the request, you can control how the response works with futures etc. Our front end webapp is using Vue cli, which clearly is javascript. I assumed the discussion here was about the server, not the front end webapp on the browser.

The discussion was about an app server written in CL (Wookie) that supports a backend app written in CL, all in async (via cl-async).

It seems to me you're using Woo as an Nginx replacement, not as a CL app server, which makes total sense to me after you described the model you're using.

I wouldn’t describe 4x nodejs as “super fast”. Nodejs is one of the slowest popular backends, or at least it was when I did a comparison a few years ago.

In the last benchmarks, woo beat all including the Go webserver: Scroll down just a bit to see the comparisons here: https://github.com/fukamachi/woo

Nice. Node has gotten a lot faster than when I last looked apparently. In fact, I’m a bit suspicious of the benchmark they’re using.

What is your distinction between "async" and "using threading at a lower level"?

In this case, an async (via cl-async) application served by an async app server (http://wookie.lyonbros.com/), so async all the way down using evented I/O vs an async server (Woo) that farms out all requests to a synchronous thread pool.

I've been asked multiple times why I "didn't just use Woo" by people who don't understand that Turtl's server was async and Woo doesn't support async.

I guess the question is what does that have to do with it? Why not just work asynchronusly with requests that hit the server with a library like lparallel with their futures? Isn't it about the response to the request?

> Isn't it about the response to the request?

High level, yes. Mid/low-level, it really depends on what you're doing. If you're serving static files, sure, use Nginx/Woo. If you're running an evented CL application that deals mostly with network i/o, threading is going to be a tank when you need a hummingbird.

I built cl-async/wookie as parallels for nodejs/express in common lisp.

Oh, are you the developer mentioned in the main post?

If so, I agree a lot with what you said!

I am! Turtl is my baby.

Awesome stuff!

The post talks about using libuv which uses event loops all the way down. I'll be honest, this whole line of questioning feels quite dismissive; OP probably knows exactly _why_ they want an event loop design, you shouldn't try to talk them into using granular thread pools.

I think you're responding to the wrong person as I didn't "try to talk them into using granular thread pools".

I asked for clarification on the distinction between "async" and "using threading at a lower level", which, for me, are equivalent things. The distinction, to the extent there is one, being that "async" often means "an existing library or language feature versus rolling it myself".

Evented ("async") concurrency, as found in Node, Python, Rust/Tokio, libuv, and Ocaml is based on building chains of events which are waited on by some fast polling mechanism like epoll or kqueue. Any IO call, say a socket read, tells kqueue/epoll to notify some handler to service the event. The flow of events drives execution.

This is distinct from thread pool models where you still block the entire thread for an IO call. While a sufficiently smart scheduler can probably then context switch out of this thread onto something else as the thread waits for an IO response, this is distinct from having the event directly wake up a handler.

That's usually what I associate as the difference between an event loop model and a threaded model. You can certainly make your threads highly granular and isolate each distinct blocking operation to its own thread pool, but it's different from actually being notified and woken up for events.

> I think you're responding to the wrong person as I didn't "try to talk them into using granular thread pools".

Yeah I think my wires got a bit crossed there. Apologies. That's what I get for being snarky while not paying full attention.

Is there something special about CL as compared to other Lisp dialects? Why not choose something like Clojure, or LFE (Lisp-Flavored Erlang) with better async/library support? Then you skip over the community aspect entirely by piggybacking off the JVM or Erlang communities.

(Another alternative would be Elixir, which is basically Erlang wrapped in a nicely-formatted version of Lisp. I.e. Elixir is to Erlang what Clojure is to Java.)

I'm sure the answer comes down to something like, the author was more familiar with Node.JS so it would be a very quick rewrite. But I feel like there are a lot more Lisp-like language choices for backend APIs before you jump to Javascript.

The article describes making use of one of the key features of Common Lisp implementations, that you can change both data and code on the fly while in the middle of an exception. I’m not certain those other lisps allow for that level of dynamism.

> Is there something special about CL as compared to other Lisp dialects?

cl is ansi standardized. it has several fast industrial strength implementations available that are self-hosted. it is both a high and low level language. it is FAST ... way faster than the dialects you named. its interactive development env is second to none

> Elixir is to Erlang what Clojure is to Java

i think LFE is meant to be that for Erlang. and i think that calling Elixir a lisp is stretching definitions quite a bit. as an asside there is an implementation of common lisp that resides on JVM - https://www.abcl.org/

> Is there something special about CL as compared to other Lisp dialects?

It has by far the best interactive development experience.

Why? Asking as someone who only knows some Scheme.

The Common Lisp debugger, in particular, is not part of Scheme (or at least the Scheme standards). CL is designed around interactivity in a way that Scheme is not and the interaction I describe next is a good example of that:

Try doing the Scheme equivalent of this CL code and REPL interaction:

  ;; in a source file
  (defun simulate (instructions)
    (let ((state 0))
    ;; this is based on an AoC problem I recently did
       (loop for (op val) in instructions
          finally (return state)
          do
            (ecase op
             ))))
  ;; in the REPL
  (simulate '((inc 5)))

Note that nothing happens in that case expression right now. It will error out and send you to the debugger. Without selecting an option in the debugger, go back to the code and change it to this:

  (defun simulate (instructions)
    (let ((state 0))
    ;; this is based on an AoC problem I recently did
       (loop for (op val) in instructions
          finally (return state)
          do
            (ecase op
               (inc (incf state val))))))

Recompile that code in the source file, and then tell the debugger to restart. Now it works and correctly returns 5 as a result.

You can also do this much deeper in a program (I actually implemented the Advent of Code "virtual machine" this way). It doesn't restart the entire program, only the offending bit.

(NB: All code written in the HN comment field, not actually tested so I may have mistyped something, should be correct though.)

To the best of my knowledge, that sort of interaction is not standard in Scheme implementations.

Basically because Common Lisp (along with Smalltalk and very few other languages) is designed from the ground up to support livecoding as its primary workflow.

The basic idea is that you start up your Lisp and it's already a working program; it just isn't the one you want to make. So you teach it interactively, a piece at a time, how to be the program you want. The workflow is less like constructing something according to a plan, and more like painting a picture or writing a document, looking at what you've got so far and making improvements and corrections as you go, all while the program continues to run. When the running program is what you want, you save it and ship it.

The ANSI Common Lisp standard defines and standardizes a collection of features designed specifically to support this workflow. Because these features are in the standard, you can count on having them in any conforming implementation.

For example, it defines a suite of language features that enable the runtime to automatically detect when you change the definition of a class, and it can automatically reinitialize existing instances to conform to the new definition. If you're not livecoding, those features make no sense; if you are, they are a godsend.

It defines a suite of error-handling features that ensure that a conforming Common Lisp implementation responds to a runtime error by creating an interactive repl known as a *breakloop* that exists in the dynamic environment where the error occurred, before the stack has been unwound, and which provides interactive features you can use to explore that environment, edit it, redefine any values, types, functions or methods, including the ones that are pending on the stack, and then resume execution as if your new definitions had been in place to begin with.

It makes absolutely everything in the runtime environment easily visible to the programmer and places it all under program control, so that you can inspect, edit, and redefine every aspect of the development environment while your work-in-progress is running, and see the effects of your changes instantly.

Those are all common features provided for by the language standard. In addition, there are other conveniences for interactive programming, such as featureful interactive editors and image saving, that are not standarized, but that are sufficiently common in the shared history of Lisp implementations that they are notable when they're absent.

These kinds of features are designed to facilitate working on a living program, modifying it interactively while it continues to run. They enable you to redefine every aspect of the program without ever being required to restart or rebuild it. Of course, that's a lot of power, and sometimes you'll get yourself into trouble with it, but the tools are designed to help you dig yourself back out, or to restart, if you must, from a convenient starting place.

I have about as much experience with Scheme as with Common Lisp. I've been using both since the late 1980s, and have shipped multiple applications and systems with both. I like Scheme. It has some particular features and conventions that I like better than Common Lisp's.

But Common Lisp is what I turn to first, and that's because it's designed around livecoding and Scheme is not.

Some Schemes and similar languages (like early Dylan) have historically provided some of these features. The only Scheme-like language implementation I know of that provided them all was Leibniz, the early Dylan development environment at Apple--and it had them because it was written in Common Lisp and inherited Lisp's features.

Full-featured Smalltalk environments are just as good. Interlisp is just as good (though very dated at this point).

Few other development environments come close.

This is the crux of the issue but it's not explained further:

> I wanted to be able to use asynchronous programming because for the type of work I do (a lot of making APIs that talk to other services with very little CPU work) it’s hard to get much more performant.

Presumably the author of Turtl hit (or predicted) a scaling bottleneck with synchronous CL? Does CL not perform very well with tons of threads doing synchronous i/o (compared to other languages)?

I also don't see a lot of upsides here for CL. The pros given are:

* You can modify code while it's running so it's useful for debugging. Is this so much more useful to warrant switching languages from a language where I just re-run my whole unit test after modifying the implementation? They tend to be imperceptibly quick.

* It's elegant, fast, and stable. Compared to NodeJS I guess I'm willing to just accept these :P

CL doesn't have a truly standard way of writing async/multi-threaded code. Every (practical) implementation supports native OS threads, and Bordeaux threads supports every reasonable implementation while providing a uniform way to access their lower level (and implementation specific) versions. You can get good async/multithread performance out of Common Lisp. But it's, by definition, non-standard. Node at least made asyncio a part of its standard implementation, so you're not having to pick and choose or construct it from lower level primitives. There are other higher level concurrency libraries for CL that build on BT and the lower level primitives to remove the need for you to think about it, but they're still non-standard.

That's all missing from CL as a part of the standard, if CL were to get a second standard something like that would almost certainly make it into the revision. But it's not getting another standard any time soon.

> if CL were to get a second standard

Huh, TIL. Checking wiki, the final/current version is from 1994[0]. Welp, that explains it for me. Thanks!

0: https://en.wikipedia.org/wiki/Common_Lisp#History

We simply use lparallel with incredible results. We combine that with using a thread-safe version of cl-containers, such that we save futures in a container, and then when needed, force the futures in the entire container.

https://lparallel.org/

https://github.com/gwkkwg/cl-containers

> With CL, it felt like I was constantly fighting the tide.... just to get basic things working that are already solved problems in other languages (Node).... I was mostly fighting it alone.

Based on the author's comments in the article, I didn't get the impression that he hit any performance bottlenecks -- rather, it was an emotionally draining experience to be on the hook for everything. Similar to how some open source authors walk away from things because it's "Too Much".

Yep that part makes sense to me. But I meant: if there's no perf bottleneck why care about async i/o at all? Since that's the impetus for async i/o usually. A sibling comment to yours answered that part for me. My understanding now is that concurrent code in any paradigm is going off the beaten path, i.e. away from standardized CL, so at the very least you aren't going to be able to write concurrent code portable across different CLs.

Async IO is not chosen because a language isn't fast enough, it's because IO isn't fast enough. Async IO would be useful even if you handwrite everything in the best and fastest assembler possible because in the time it takes to go out and read some data (even on an SSD) you can compute many, many more things. The purpose of async IO is to permit the interleaving of fast and slow activities. If I were still doing scientific computing (it's been a while) I'd want async IO (some implementation of it) even if I had the best GPU for my number crunching bits. Loading the model takes a lot of time and there are other activities the program can engage in during that delay.

CL's lack of a standard way to do async IO (in this particular instance) means that even if you have a very fast program, you end up with either a bottleneck waiting for IO or you've recreated the async IO model from other languages (or their standard libraries or their de facto standard libraries) and are now responsible for it.

> The purpose of async IO is to permit the interleaving of fast and slow activities.

You absolutely do not need async IO for that. You can do that with a synchronous programming model just fine, using threads and letting the OS do its thing.

Async is an absolutely horrible programming model. There are really only two advantages is brings: It lets you avoid the memory and context switching overhead of threads if you need to handle a very large number of requests concurrently, and it lets you do concurrency without worrying about synchronizing access to shared resources (but there are much better ways to do that).

If you use threads to cordon off the slow IO code, you're doing async IO, just without core language or library support for it.

I was referring to using blocking IO, one thread per request. You know, the much simpler way that was the norm before async became fashionable.

> You absolutely do not need async IO for that. You can do that with a synchronous programming model just fine, using threads and letting the OS do its thing.

"just fine" is highly relative here and is doing most of the work in your statement. Do you have any conditions under which this is "just fine"? Any more detail? Or else it's just a personal preference.

A preference? The term was mostly meant to convey that it can be done without problems, contradicting the assertion that async IO "permits" the interleaving of fast and slow activities.

Synchronous models have been used for that far longer and more often than asynchronous ones. CGI scripts, Perl, Java servlets all did synchronous IO while also interleaving slow and fast activities for decades before Node made async IO fashionable.

An in my second paragraph, I specifically mentioned the specific conditions where async has advantages.

> Synchronous models have been used for that far longer and more often than asynchronous ones. CGI scripts, Perl, Java servlets all did synchronous IO while also interleaving slow and fast activities for decades before Node made async IO fashionable.

Node didn't make async IO fashionable, as much as you seem to want the beat the drum of "new/hype" vs "old/Lindy/mature". Synchronous IO was slow, memory-inefficient (to spawn multiple threads), and didn't scale well. Slow enough that the C10K [1] problem was framed to capture the issues. Event loops in net servers were mostly popularized by nginx [2] to explicitly solve the C10K problem, which is what started driving folks to use event-loop async programming. Moreover event loops had been popular for years in GUI programming before multiple cores simply because CPU-level parallelism just _was not possible_ (well except for ILP which is a bit different). For example, Tcl/Tk had an event loop driving GUI display logic for ages [3], which really is the same problem. Instead of waiting on NIC events, we were waiting on keyboard/mouse events instead.

Just because it's old doesn't mean it's good. There are lots of old bad things and lots of old good things, just like there are lots of new bad things and lots of new good things.

[1]: https://en.wikipedia.org/wiki/C10k_problem

[2]: https://en.wikipedia.org/wiki/Nginx#HTTP_proxy_and_Web_serve...

[3]: https://wiki.tcl-lang.org/page/Tcl+event+loop

> Node didn't make async IO fashionable

As far as I can tell, Node made it fashionable to use the async programming model for the entire application in server code. I'm sure it was done before, but not nearly as commonly, especially not to the point that people, like the commenter I originally replied to, consider it a prerequisite to "permit the interleaving of fast and slow activities", even for scientific computing.

> Synchronous IO was slow,

Not fundamentally though, but because of implementation details like bad thread scheduling algorithms.

> memory-inefficient (to spawn multiple threads), and didn't scale well

Yes, and I explicitly acknowledged that in my original comment. But it's efficient and scalable enough for most applications (which don't have millions of users) and is still very widely used, and people have found ways to make it more memory efficient as well.

> Event loops in net servers were mostly popularized by nginx [2] to explicitly solve the C10K problem, which is what started driving folks to use event-loop async programming.

SO nginx came out in 2004, node in 2009. Did the 5 years in between really see a lot of people writing asynchronous application code? In what language? It's quite possible that that was a development I missed at the time.

> Moreover event loops had been popular for years in GUI programming before multiple cores simply because CPU-level parallelism just _was not possible_ (well except for ILP which is a bit different). For example, Tcl/Tk had an event loop driving GUI display logic for ages [3], which really is the same problem. Instead of waiting on NIC events, we were waiting on keyboard/mouse events instead.

Yes, event loops are the standard in GUI programming almost everywhere. Java Swing did that too - but with an otherwise entirely synchronous multithreaded programming model.

But the reason for that is most definitely not that "CPU-level parallelism just was not possible", nor has it anything to do with "waiting on events" necessitating an event loop as your entire programming model. The reason is that multithreaded GUI toolkits have been repeatedly tried (all the way back to Xerox PARC, long before multicore hardware) and found to lead to insurmountable issues with deadlocks: https://web.archive.org/web/20160402195655/https://community...

Yeah, admittedly I could have gone with threading/hunchentoot and been much better off. It was probably stupid to forge the entire async path the way I did.

To be fair, Turtl these days gets enough traffic where async does probably make sense for us, and nodejs is serving us well.

If you gain nothing from livecoding, then no, Common Lisp's pros are not going to be compelling.

If you gain something significant from livecoding, then Common Lisp offers a good deal more than just quick turnaround time. I listed some of what it offers in another comment here.

Why Node over Elixir? Even in 2019 Elixir was much better technology for building web services than Node and this haven’t change since. With Elixir you get the CL interactive development and metaprogramming (not as powerful in CL) but also scalability.

> however when things did go wrong, there was nobody to run ideas by and nobody to really help me…I had built all these things myself, and I also had to be responsible for fixing them when they broke.

You are responsible for your supply chain. I’ve had instances at work where some rando open source library would be bugged, and we’d have to patch it up ourselves. It is much easier to fix code you’ve wrote.

> You are responsible for your supply chain.

Exactly, which is why I abandoned my supply chain! I bit off more than I could chew with CL/async.

I think what the GP is getting at is, that you are also responsible for whatever else you pick instead of CL/async.

On this:

> I don’t have to deal with quicklisp’s “all or nothing” packaging that doesn’t support version pinning.

it's easy to clone a given version on quicklisp's local-projects, but we also have Qlot and the emerging CLPM. See also the new https://github.com/tdrhq/quick-patch/

I made great use of local-projects when Turtl's server was written in CL, and at the time I'm not sure if Qlot/CLPM existed (never heard of them until your comment). This was many years ago the switch happened.

The first link is to https://turtlapp/ which is broken; I think it should be https://turtlapp.com/

Oh my. Thanks, should be fixed in a minute.

I thought they switched from Lisp to Rust. Was that process cancelled?

The backend switched from CL -> Nodejs, the frontend switched from JS -> Rust core bundled with JS ui.

There was an experimental app core written in CL (https://github.com/turtl/core-cl) that is basically the precursor to what the rust core is today. I did have success embedding lisp (via ECL) into other components (like node-webkit at the time) but ultimately it was kind of an uphill battle, as with many off-the-beaten-path things in common lisp.

tl;dr three major pain points:

(1) no native async

(2) small library ecosystem

(3) CL community NIH syndrome:

> I think the straw that broke the camel’s back was when a few people started making copycat projects that added no real value (other than benchmarking fast) but stole mindshare from all the work I had put in.

These are good points. (1) is a real, true problem with CL (as opposed to things that people commonly think of that aren't actually that bad e.g. syntax), and it's probably not going to be solved in a satisfactory way anytime soon - the design of the language is pretty nice, but seems to exclude first-class continuations (according to [1]), async, and coroutines. (yes, you can emulate these things using macros, but that often involves tree-walking, and results in non-orthogonal features that don't play great with the rest of the language - first-class is always better)

Common Lisp was designed for, and is very good at, representing algorithms under a very specific model of how the world works - that is, single-node and single-memory-space. Its design assumptions don't hold up very well in a multi-node, concurrent world. Yes, there's bordeaux-threads and lparallel, but they feel like hacks, and the presence of GC means that they won't be able to attain the performance of Rust or Erlang (which seems like Lisp for a distributed world - its model of the world seems to be fundamentally distributed, not a bunch of patches on top).

(2) is somewhat unavoidable when you have a small community, but (3) is not (and is highly undesirable). Ideally, the CL community would document all of their published code really well (from the library-author side), and then put some effort into learning how to use (and improve) someone else's code (from the library-user side) instead of rewriting their own 50%-complete version of an existing library, but that would require a significant shift in the attitude of the average CL user, and I'm not sure how to try to affect that.

[1] https://stackoverflow.com/questions/16651843/why-doesnt-a-pr...

Yeah the CL community likes to be lone warriors, I am definitely very guilty of that.

Documentation in CL is pretty bad. Worst is when people say its well documented and its nothing but, all those manuals just listing all the functions are a complete nightmare and useless to me

I'm guilty of this too! Change begins with one's self, they say, heh. (I should learn to write good docs before castigating the community)

And you're right - trivial docstrings for all the functions isn't documentation at all. Real documentation is the four-part system[1] of tutorials, howtos, examples, and reference material - I've found this mental framework to be enormously helpful for both (a) identifying when some other documentation is inadequate and (b) improving my own.

[1] https://documentation.divio.com/

That is why I've put about half of this year into the Common Lisp documentation generator for all of my libraries.

If you are interested, please read it's docs and join the effort of making good documentation for CL projects: https://40ants.com/doc/

Thanks, will check it out!

Agree 100%

I think (2) should be more "small ecosystem of ASYNC libraries", that's what he complains about.

Hey guys, don't miss this nice interview with Kina Knowledge: https://lisp-journey.gitlab.io/blog/lisp-interview-kina/ (https://news.ycombinator.com/item?id=28961987)

Atwood's Law in action.

“Any application that can be written in JavaScript, will eventually be written in JavaScript.”

(https://en.wikipedia.org/wiki/Jeff_Atwood)

Reply on news.ycombinator.com