Ask HN: What's the deal with Common Lisp?

(Note, I myself started with Lisp prior to Common Lisp by a few years, but I used it near its inception and have vaguely followed it since then (I soon switched to Scheme and now am playing with Clojure).)

"Have I just hit a bad corner of the Lisp world?"

Yes, you have; for the things you want to do you should try either a Scheme that has the libraries you need (I'm not sure if there are any (probably not, but maybe try one that e.g. already has IRC stuff?)) or Clojure, which by virtue of sitting on top of the JVM and having fast and clean interoperability handles the libraries problems.

Dependencies are handled by existing parts of the Java ecosystem and/or the relatively new Leiningen system (the name is a play on ant, the first Java build system).

Warning, in terms of functional programming, there's a continuum of CL < Scheme < Clojure. Unlike the others, Clojure is not multi-paradigm, you will use it functionally (well, unless you use it as syntactic gloss for programming in Java, which misses much of the point).

Here's the most recent HN item on getting up to speed with Clojure: http://news.ycombinator.com/item?id=1086562

Note that getting SLIME up at this very moment might require a bit of trickery, a recent SLIME commit broke it for Clojure's use of it. Be sure to check out the Clojure Google group and feel free to ask any sorts of questions on it if you get stuck (the community is a lot better than CL's).

(Historical notes: CL is bad for networking and concurrency because it was formalized before there were standards for those areas and it has been stuck in amber since then. Various implimentations have non-standardized libraries for them (there may be something of a standard for threads, I forget, but in many CL implementations threading support itself is iffy at best). And you already know the networking story (which CL are you trying?).

Scheme standardization is very slow; it is progressing, there are optional SRFI (Scheme Requests for Implementation) standards for each of the above (I think, not sure about networking), but I can't recommend it for what you're trying. If you e.g. wanted to start from the bottom up by going through the SICP or another book, that would be fine, or if you wanted to research language stuff it's great, but it just doesn't yet have a general modern programming ecosystem and won't for some years if ever. In general, you match your needs with the implementation and its packaged libraries and go from there. If you want to try that route anyway, ask me for some pointers.)

Bad corner of the Lisp world? Socket libraries are the Skid Row of Common Lisp.

The reason it's hard to create a good library comes from an interplay of 3 things: streams, lack of standard character representation, and lack of standard foreign function interfaces. The latter two are sort of advantages in that if they were standardized in 1989 or whatever, we would probably have wound up with something stupid involving DOS code pages and Fortran on VMS.

Streams are convenient and most CL functions expect them. They're also a really inefficient way of using BSD sockets. Especially when your Lisp implementation's character encoding isn't UTF-8. So if you want to write fast network code in Lisp you basically need to use the raw POSIX FFI functions and byte arrays.

Then you get into fun stuff like vectored I/O (which you may or may not be able to do depending on whether your GC moves arrays around), and event-based I/O (which people disagree as to how to do properly). But it's ok, because none of the existing socket libraries try to support this anyway, so you're back to calling POSIX functions via FFI.

(I gave a presentation about this stuff last year: http://carcaddar.blogspot.com/2009/11/november-montreal-sche...)

If your socket library is dependent on any third-party libraries, pray that the latest versions will not completely change their interface and make your library fail to compile. Then multiply all this by the half-dozen or so Common Lisp implementations people expect to be supported.

Just like Skid Row, this has been a recognized problem for many years (well over a decade for certain), and it's kind of hard to make it better. Flexi-streams over usocket is the best off-the-shelf solution, so you're probably doing the right thing.

I'd appreciate hearing from other people who came in, were similarly disgusted, but have come to understand the whole thing better.

Ok. The larger CL world is terribly fragmented. There's a smaller world inside the larger world which is really good. The trick for the beginner is distinguishing between the two, i.e. between what is detritus to be ignored ("massive twisty world of broken dependencies and half-finished libraries" describes that perfectly) and what is the real stuff (e.g. Edi Weitz's libraries like flexi-streams). Once you find your way to the smaller world, the fragmentation and dependency issues don't go away, but they become minor and occasional relative to the mind-blowing amazingness of what CL is good for.

How do you find your way to the real stuff? Trial-and-error and asking questions. Is that an acceptable answer? No; many people unfortunately never get over the confused/disgusted stage and bail. Does it work? Yes, very well, if and when you get over those initial hurdles.

I fully agree that the above situation is upsetting when you compare it to things that just work out of the box. There's so much half-baked and half-working stuff in the CL world that I can only compare it to spam. A lot of Lispers are unhappy about this, and many attempts have been made to fix it, but it isn't quite clear what a definitive fix would require.

My personal hope is that the work being done by the Clozure-with-a-Z people (CCL) will blossom into a well-functioning, self-sufficient, obvious place for people to begin with Common Lisp. They do seem to be getting there. Importantly, they have the nucleus of something that CL has been lacking for a long time (since, I suppose, the implosion of comp.lang.lisp into various forms of nastiness): a nice, normal community.

I'm no expert in any of the Lisps, but find that the learning curve with CL is massive. Of course, the return on your investment in learning is huge as well.

Depending on what your applications are, you might find something that really hangs together well, like newLISP a better fit. And it's very simple to intergrate libraries written in C.

I'm a bit concerned by any Lisp that still uses dynamic scoping (Emacs Lisp is the only other current Lisp dialect that does). I gather that newLisp tries to mitigate the issues, but I'm not sure how well.

"Have I just hit a bad corner of the Lisp world?"

Sort of.

The real problem is that it is difficult to know what the current 'best' implementations of different libraries are, and what versions of common lisp they are intended to build under (and operating system!)

This is compounded by the fact that the higher level libraries have more dependencies.

As there are dozens of libraries and a smallish CL community, this leads to the problem where author of sub-library a makes change which breaks dependent library b, and no one realizes this for months and months. When someone such as yourself comes along (even if it is possible for the library to work quite well with minor changes), the build process fails is not so obvious about what is wrong. Your reaction is to say 'CL is totally fucked' and run screaming to the hills is only natural.

That said, I have a few tips:

1.) If you are going to use an open source lisp, linux is generally the easier operating system to run it on.

2.) If you are going to download a library, don't use asdf-install.

ASDF itself is great for actually structuring the project and laying out your directories in a coherent manner (managing dependencies), but asdf-install is a waste of time (in my opinion, anyway, I'm sure someone will disagree with me).

Instead, go to the library's page/repo and download the latest version.

Make sure that someone has been working on the library recently (I try for < 1 year since last update). (Another good way to check if a library works is to search Github... EX: http://github.com/search?type=Code&language=common-lisp&... ).

Once you get a version that works, hang onto it somewhere in case the guy working on it breaks it with an update. If you can find a version that almost works, it is (a lot of the time) worth it to salvage it and push the updates.

Just read the error messages very carefully and it should be fixable.

asdf-install is great in theory, but there are too many synchronization issues, and not enough humans pushing fixes/updates to manage them.

3.) Keep your lisp image up to date. The people developing libraries are generally using the latest build of clozure/sbcl/etc. You should too.

Scheme implementations don't have these problems because there is no expectation of write once run anywhere with regard to libraries. Different implementations have their own package managers and BDFLs.

Clojure doesn't have this problem because it hasn't been around long enough to have a lot of dead libraries.

I'd like to point out that it is perfectly reasonable to program functionally in common lisp. I take a fairly functional style in most of my code.

edit:punctuation, grammar...

"I'd like to point out that it is perfectly reasonable to program functionally in common lisp. I take a fairly functional style in most of my code."

Absolutely; my point on FP WRT these three Lisps was that you don't have any (realistic) choice not to go all-in functional with Clojure, whereas Common Lisp is the most multi-paradigm and you can get away with an imperative style should you so choose.

I don't know mcuh about LISP. But what I know about OSes means I know the advantage of Everything is a File.

On Plan9 if you can open / read / write / close / remove you can do almost anything : networking, graphics, serial, debugging, mouse control, the list extends by design.

We have a few Schemes ported for those who had a personal itch. ALISP claimes a port but their site is down for me.

I have a few Schemes installed on my Plan 9 system, but I haven't really done anything with them. I'd really like to see an example of integrating a Scheme with devdraw; I've seen everything else in your list done by file operations but only used regular library calls for graphics.

Forth is in the same boat, no-one is going to make C based Plan9 libs for it. The one I use 4th only has file ops anyway so I plan9 to do something with it but it is the perennial todo list problem :)