Comment by orthecreedence | Hacker News Reader

Comment by orthecreedence | Hacker News Reader

Parent Full thread

I wrote some high-performance lisp applications in production. My main was was a threaded queue worker, driven off beanstalkd, that processed large volumes of data. This was highly successful and didn't require overly-provisioned servers (was using CCL/linux x64/linode VPS).

That said, I don't think I'll ever use lisp for front-facing web stuff again. I wrote cl-async, and afterwards and used it in production in a few places, then a bit later started using node heavily. The tooling for CL is just not there. Quicklisp is great, but not being able to pick and choose package versions is limiting, and having to rewrite every single thing for async was a pain (since CL doesn't have continuations, any sort of async that avoids CPS is impossible). I was hoping I would get more community support on cl-async, and I did get many hands helping, but eventually it just became too much to maintain. So if you're doing network stuff on CL, it's probably best to just use threaded, which IMO just doesn't handle the same amount of load evented does, at least when dealing with web APIs.

One thing I used lisp for briefly which I want to do again soon is game dev. Being able to redefine/fix portions of your program while it's running is fantastic, especially when you have huge amounts of state to deal with like you do in games. I wouldn't exactly call that running CL in production, though.

FWIW, Qlot solves the version pinning problem. Not to discredit your experience, just wanted to say that there are solutions today for the same issue.

I'm interested exactly what the pain points are with CPS. Do you mean manually rewriting in continuations is tedious or is there a more insidious issue. I haven't tried writing async code in Lisp but macros would appear to eliminate continuation-writing-by-hand.

> macros would appear to eliminate continuation-writing-by-hand

They don't =]. There's cl-cont, but it's like building a house on a foundation of toothpicks. Not to discount the work the author put in, but you can only do very specific things with it, and last time I looked, error/exception handling was pretty much completely missing.

> I'm interested exactly what the pain points are with CPS

Basically, for example, if I'm going to convert cl-postgres to use async, I have to rip out the protocol parsing from the networking (they are often globbed together) then I have to replace the networking with async and tie it back into the protocol parsing. This in itself isn't too bad, but then anything that calls the new async cl-postgres has to be converted to expect async...all the way up. This can be somewhat alleviated by promises (https://orthecreedence.github.io/blackbird/) but they still don't change the fact that the entire stack is unwound once you call into async.

Continuations would allow building green threads/coroutines which would eliminate CPS and allow you to preserve your stack while calling into async, but CL is pretty much written in stone to not have continuations (or coroutines et al) at this point.

Qlot looks handy, thanks for the tip.

> then I have to replace the networking with async and tie it back into the protocol parsing. This in itself isn't too bad, but then anything that calls the new async cl-postgres has to be converted to expect async...all the way up.

Isn't this the very consequence of going async? I'm reminded of the article on this recently featured here[1]. Once you introduce async, you poison everything up to the event loop (which, with usual async implementations, you don't even see or control).

[0] - http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y... + https://news.ycombinator.com/item?id=16732948

> Isn't this the very consequence of going async?

Like I mentioned, depends on the language features. If it has coroutines/green threads/continuations, you can use async without unwinding the entire stack. CL will probably always have this problem, unless the bigger implementations independently decide to implement the necessary building blocks (which they probably won't).

Try explaining that to me 5 years ago, though.

If it's about avoiding stack unwinding, then maybe you could hack an implementation with the condition/restart system? This is the one stack-preserving feature in CL.

Biggest issue I personally had with CPS and why I'll never use it for anything is the need to wrap your entire call stack, at compilation time, in CPS macros. That's a no-go for calling external libs.

Clojure does this a lot better with Pulsar.

Thanks for your work on cl-async/blackbird: we also used it in our recent project (and you should have gotten a couple of related requests)! I don't think you should be discouraged though: your contribution is very valuable and surely was noted, just that the whole async approach is, in my view, not in line with the general direction of where Lisp is most applicable, currently (we should admit that it's a niche language). After all, having an evented server means that it doesn't perform a lot of useful computations, most of the work is just serving some kind of existing content. I.e. you could say that the application logic is marginalized and commoditized. And Lisp is the best choice for the opposite cases, those that require and emphasize elaborate application logic and intense computation...

I use cl-async and want to thank you for it.

Anything further from your experience and retrospective working on the project would be invaluable to document somewhere for the CL community at some point. Just your insight here in the above comment is very interesting. Over time things may improve.

Thanks for the nice note! I'm glad you still find it useful.

I do hope things improve over time. I'd love to come back to CL, and actually plan to sometime soon. I'd like to get my pet project, Turtl (which was using cl-async/Wookie in production and is now migrating to node), in a stable place (and hopefully making money from it at some point) and then start dipping my toe into game programming again. I'll almost definitely use CL over anything else. Maybe I'll write the GPG-verifying, version-pinning quicklisp wrapper I've always wanted while I'm at it =].

As far as retrospective, I think I suffered from the part of the lisp community I had always hoped wasn't true (or hoped I could change): people are much quicker to build their own version than to improve on something existing. I think if Wookie had gotten more attention and cl-async had had a few more drivers written (postgres, mysql, etc) it could have gotten a foothold. It didn't though, and I got tired of hacking through the digital reeds with my machete when I already knew javascript pretty well and node had all this stuff figured out already. It's funny, after using Express for ~4 years, I keep having the urge to go back and implement what I've learned into Wookie. I have to tell myself that it's probably best to just let it sit for now.

On another note, my dog, Wookie, who the project was named after, died last week. So it seems even more appropriate to let the whole thing rest. I might re-use the name if I come back to the CL world, though. Seems its doppleganger project got more focus anyway.

About picking and choosing package versions; isn’t it straightforward to keep whatever versions of packages you want under ~/quicklisp/local-projects ? Or is there a need for a more sophisticated tool for this?

This works, sure, but when setting up a new server, you're going to have to remember which package version you want for each of your dependencies. It's much easier to just have a Cargo.toml or package.json and run one command that verifies and installs exactly what you need.

I guess containerization solves much of this problem now, but back when I was heavily into CL, containers were just starting to peek their heads out into the world at large.

Solaris and openvz containers have been around since 2005 or so.

Solaris is not that accessible to most people the way Linux is. And containers are overkill when all you need is the ability to put things in a sub-directory.

I'm a little bit disappointed that you've decided to stop maintaining cl-async, although I appreciate the difficulty of maintaining such a project.

Hopefully it's still useful! The nice thing about CL is that it never fucking changes so you can write something and 25 years later it still runs. Really, the only moving targets are the bundled ASDF version implementations ship with and libuv itself. I've had some reports of cl-libuv not compiling against the latest libuvs, but that shouldn't be too hard to fix (I still merge PRs =]).

All in all, I think I just hit one too many problems using cl-async/wookie in production to justify using it vs Node which just kind of works and has a plethora of maintained packages.

Reply on news.ycombinator.com