How to write Common Lisp in 2017 – an initiation manual

Curious over why CL vs Clojure? Any comments.

Which lisp implementation are you using? I had the pleasureof using AllegroCL at a gig I did recently. The environment has a bit of a learning curve, but once everything clicks it is like heroin.

I ended that job a couple of months ago, but I still have wet dreams about it.

which lisp interpreter did you use ?

Can you elaborate on your use case?

How do you sneak something like lisp in to work?

Neat. Does Roswell do all this without modifying your current lisp/emacs/slime/quicklisp if you already have them set up the way you like but want to play around with it?

And is there something like Roswell for Clojure?

I mentioned elsewhere I'm learning Common Lisp. I'm also learning Python by translating some Common Lisp code into Python and part of that is to make sure I understand what the Common Lisp is doing. As an understatement, that's meant building some very unPythonic abstractions (yay, me).

Anyway, I think there is a fundamental design difference between Common Lisp and other 'first class' programming languages: Common Lisp was designed as a way to use computers because a computer user would sit down at a Lisp Machine (which was the future when Common Lisp was designed) and use Common Lisp to do ordinary computer stuff like store their recipes and manage appointments and copy files between directories. It's reflected in the :cl-user package not being called :cl-programmer and the logic of typing (in-package :tps-report) on Saturday at the usual time.

Most other languages don't have this idea...In Unix users don't fool around with stdio.h. In Python, there's no clean way of switching between applications at the REPL -- or rather interpreter -- because of how Python handles the hard job of naming things (like most languages, there's a bit of punting on third down with the catchall epithet 'unpythonic').

This makes it hard to get one's head around Common Lisp, but it explains why a person might leave a REPL running for a week and make better progress because of it.

As other people have said, you can use Lisp the same way you describe using Python or the same way you describe using C.

You also have a third option: you can use Lisp as a sort of interactive command-line calculator-plus-kitchen-sink. Leave it running and teach it to do odd jobs for you. If you accumulate your odd jobs into a file then you'll have them for later. You could of course save the state of the running Lisp, but a source file is better in that it keeps a nice tidy record of the source code of your hacks.

Serious programs written in Lisp are mostly organized pretty much like serious programs written in other languages: the program is factored into a collection of source files that, in the best case, reflects a logical decomposition of the functionality. There's some form of system loader that compiles and loads the sources in the right order and, if desired, dumps the result into an executable program.

Nowadays most people use ASDF for system loading and dumping, I think, but it's not hard to write your own system loader, and I still know people who prefer to do it that way. Just as an example, CCL still uses its own homegrown loader to build.

If your program is like a Python script then it's a Lisp source file that you pass to the Lisp kernel, in just the same way you would do it in Python.

If it's a compiled executable, then your sources were compiled into the Lisp and dumped to that executable, and it works pretty much like a C program. The main difference is that there is no distinguished main(); instead, Lisps generally treat the Lisp's own repl as the default main function, and the image-dumping tools offer you the option to substitute the main function of your choice.

Things are a little different, but only a little, and it's pretty easy to learn how they work.

In a compiled system like SBCL, things work much as they do in C, except the process is more "programmable."

To recap: when you load a (dynamically-linked) C program in Linux or OS X, the OS will create a new process, and map the dynamic linker (ld-linux.so or dyld) into that process's memory. It then transfers control to the dynamic linker. The dynamic linker then loads the rest of the program into memory (from ".so" or ".dylib" or ".dll" files) and links everything together (i.e. resolves symbol addresses) before passing control to the application entry point.

A compiled Lisp system like SBCL works much the same way, except it uses its own linking/compilation machinery. A new process is bootstrapped by running a C program, which loads the Lisp image into memory. The image is compiled code/data containing the Lisp runtime, compiler, and standard library. After the Lisp image is loaded, you've got two things to work with: COMPILE and LOAD. COMPILE takes Lisp source code and compiles it into a FASL (compiled code and data, equivalent to a ".o" file). LOAD functions much like dyld or ld-linux, and loads compiled code and data into the running process. Indeed, in some implementations like ECL, LOAD is just built on dlopen(). The "eventual program" is ultimately built by LOAD-ing the FASL files comprising the program.

In practice, you don't do this manually. Instead, you use something like ASDF: https://common-lisp.net/project/asdf/asdf.html#Defining-syst.... ASDF functions much like 'make' in that you've got a system definition listing the files belonging to your program. Whenever you call ASDF from the REPL, it'll look at what source files need to be (re)-built, compile them to FASLs, and (re)-load them into the running process. In contrast with C (but similarly to Python), Lisp allows code to be loaded into a running image. In the Lisp workflow, you don't restart the program you're writing each time you recompile (although you can, if you want). Instead, you keep it running, and load or re-load code into it as you work on it.

Common Lisp implementations do whatever you want.

> I'm used to languages like Python, that have a number of files that are modules, and to start a program you run one of them as an entry point.

You can do that. start a Lisp with a file to load at start, it can then load the dependencies.

> C programs consist of a lot of files that are compiled and linked into a binary executable.

You can do that, too. That's typical when you create a 'system' declaration which describes your software. ASDF would be a tool for that. You then compile the software, load it and dump an image. Some implementations have a more elaborate way to create applications or can create loadable libraries, which you can integrate into other applications.

> You build an in-memory state by adding things to it, later dump it to a binary. How do you get an overview of what there is?

Typically you would work with files. Write the code in files, evaluate the code from there and build the software from time to time as a whole. If you use SBCL, then you get tons of compile time information, type checks, efficiency hints, warnings, ....

You can build your Common Lisp programs from a set of files like any other programming language. "asdf" is the most common system which provides loading complex lisp system. I actually use a makefile to build my Common Lisp based executables from my set of files, no neeed to build up an in-memory state.

The difference with Common Lisp is, you are not limited to this build model, but while your whole code is loaded, you can keep redefining functions for development. This creates the minimal cycle of editing and testing any single function in your program. This is a big asset in development, but in no way means you have to build your systems that way. Actually the contrary, I can only recommend to restart your lisp image frequently to ensure the system loads and builds from file rather than depending on the image state.

I don't know where you got that impression, but CL development is no different than Python development.

Code goes in .lisp files, and then you either load it into the REPL or pass it to the compiler and tell it what the entry point is.

I suppose it's technically possible to develop an entire application at the REPL and then dump to a binary, but I don't think anybody does that any more than they do it in Python.

Lisp offers a lot of ways to check what there is. You can inspect, search, look at source code, docs, assembly code. All of these things are dynamically inspectable. Tools like Emacs and SLIME make it easier.

You can still look at the files that get loaded. Lisp organizes itself around systems (libraries) and packages (namespaces). It's good to check what packages a system has, then you can check out the symbols provided by a package.

Lisp isn't totally wild-west in the concept of the Lisp image. There is organization to good code.

In Common Lisp, you write to source code files and then use ASDF/Quicklisp to compile/load that project. If you feel the need to create a standalone executable you can dump the image with an entry function specified. It's essentially the same as python, although standalone executable are less prominent than in-image programming.

This talks about GC settings that worked for them in SBCL for production, not sure if you already tried the same kind of stuff but may be an interesting read http://tech.grammarly.com/blog/posts/Running-Lisp-in-Product....

If you can make an easy to run example of the garbage collector not working correctly, the folks in #sbcl on freenode are very responsive and have commit access to fix it.

You might want to mention https://github.com/fare/asdf/tree/master/uiop which is part of ASDF and bundled by default with most implementations.

Also, the ASDF information is somewhat dated. It will work, but the central-registry has been deprecated for years. I'm not even sure how it might work on windows.

If you're recommending quicklisp, there's the local-projects approach, otherwise there's $HOME/common-lisp/

IMO the "Getting Started" section presents way too many choices. Perhaps instead just link to Portacle (or some other lisp quickstart) with a mention that there are other options?

> Why hasn't anyone made a more eye-frendly version of the Common Lisp Hyper Spec ? Having good, easily-browsable documentation is a core-problem.

A friend of mine is working on that as we speak. The CLUS, or Common Lisp UltraSpec:

https://phoe.tymoon.eu/clus/doku.php

That said, CLHS isn't bad - it's lightweight, available off-line, and on-line you can pretty much always find what you're looking for by searching for "clhs [term of interest]".

> some libraries had a compatibility matrix... with common lisp implementations. that seemed weird to me.

Common Lisp has a pretty good standard, but it's a bit old so it didn't predict some things we're currently using, and also left some other things to the implementers. So some features are only available via vendor-specific extensions, which makes it necessary for some projects - especially compatibility libraries - to be tested across plethora of CL implementations.

The flip side is that you have a few commercial and open source implementations to choose from, which is valuable if your project can exploit strengths of a particular one.

> Why hasn't anyone made a more eye-frendly version of the Common Lisp Hyper Spec ? Having good, easily-browsable documentation is a core-problem.

The hyperspec is only distributed under a restrictive license. The TeX sources of actual last draft of the ANSI specification is public domain though. This means that someone would need to retranslate from the TeX sources rather than modifying the hyperspec.

> Why hasn't anyone made a more eye-frendly version of the Common Lisp Hyper Spec

I mostly access the HyperSpec using Erik Naggum's hyperspec.el while programming in Emacs, configured to bring up Lynx. A lot of other people probably do something similar, so there has not been much interest in adding CSS.

> The relation between the various native data-types were quite unclear to me.

I'm not sure what was unclear. Perhaps you cuold give a specific example of something that confused you?

> * The relation between the various native data-types were quite unclear to me.

http://sellout.github.io/media/CL-type-hierarchy.png

It's a seamless dynamic programming language, which can, with care, be given excellent performance and a high level of abstraction. In my opinion, it's miles better than the other dynamic languages out there, by nearly every factor. It rewards investment and development very well; it's a tool for mastery, not for quick and easy starting.

If you're looking for statically typed languages, it's not going to win there. But my experience writing a lot of Perl and Python, with a certain amount of helping with Clojure and Ruby experience, strongly indicates that Common Lisp is very very competitive there outside of the 'library' front.

There are some other practical answers here so I'll take a different angle.

Fun! CL is a language to play in, after a day of wrangling Java & ObjC issues I love settling down to just play in an environment that lets blast some code out and play with ideas. Of course this applies to other languages too and this is dependent on your interests, so the case I want to put out there is:

Even if a language isn't suitable for your current business needs, see if it gives you joy. Languages have trades offs to meet their goals, evaluate languages for pleasure too.

Also come visit #lispgames on freenode sometime..most of us are procrastinating making engines but it's always nice to have new folks around.

I'll take a stab at this glib comment:

1. You've inherited an application written in Common Lisp

2. Common Lisp has features you find desirable that aren't available in another system

3. You like Common Lisp

4. Common Lisp helps you get the thing you're trying to do, done

This is a question that has been answered many times over the years. Search for "why common lisp" and you'll get a long list of them.

The answers now, IMO, are not substantially different than they were, say, ten years ago. The big differences, I think, are the new competitors in the LISP world: the rise of Clojure, and the rebranding and further development of PLT Scheme as Racket.

I did a presentation about this some time ago where I try to explain how I develop Lisp software. Perhaps someone will find it informative:

https://engineers.sg/video/web-development-in-emacs-common-l...

Well then you are giving us a fantasy scenario. Because in reality, the alternatives are not great.

It is. https://github.com/articulate-common-lisp/articulate-common-...

Yes, send in PRs if things significantly change! It's a stable ecosystem, but things do improve over time. Things like examples would be quite useful.

Is there something like paredit available for CCL's IDE?

1. Racket is a multi-paradigm programming language. It has Java-style class/object system, a CLOS-like object system(swindle) and a prototype object system (like self and JavaScript). 2. The macro system is arguably the most sophisticated available. 3. Functionsl programming! including '(purely)Functional Data Structures'. 4. Parallelism (futures) 5. concurrency 6. Contracts 7. Typed Racket 8. Pattern matching 9. Modules 10. Units 12. Pattern Matching 13. Exception, continuations 14. Reflection 15. arguably the most sophisticated tools for creating full languages and DSLs 16. Datalog language (like in datomic) 17.an amazing IDE - it has a tool to debug macros! (but you can also use Emacs) 18. Typed Racket 19. Functional pictures (pict) 20. OpenGL 21 cross platform (win, mac, Unix, Linux) it will even run on a raspberry pi 22. Amazing community 23. Math library (and plot library) 24. Scribble/@-expressions (like markdown but much more powerful) 26. Quite a few libraries

I think you will be happy with either one. It's targeted mainly toward people familiar with them, but here's a recent comparison:

http://fare.livejournal.com/188429.html

Note that a lot of what Faré likes about racket is somewhat intrinsic in there being one implementation, and a big part of why racket branded itself away from scheme (it was formerly PLT-Scheme).

In lisp there are still a lot of very different implementations in use, so if you want to "grow down" you either have to be non-portable or do a lot more work.

I love Common Lisp, so I will say "learn Common Lisp" but you'll probably be just as happy if you flip a coin (and I would recommend doing so rather than debating much as learning the "wrong" one now is probably better than learning the "right" one in the future).

My personal, subjective, feeling is that the Racket offers an easier entry path (docs, libraries and library discovery, relative non-cruftiness, package management via raco, IDE, etc.). I also like that Racket is a little more biased toward FP than CL. I'm told the macro system is ahead of CL's, though I can't say I deeply grok macros.

The Racket community always feels friendly and welcoming to beginners, which is something the CL community hasn't always been.

I do like that CL has a standard with multiple implementations. That said, the standard feels old, and you can quickly run into libraries that were built with less-than-universal compatibility. Things like tail recursion in CL exist commonly in implementations, but not in the spec.

I've tried to carve out a bit of hobby time for lisps over the past few years. I started with CL and fought with the tooling and I could see the power, but I never felt great about my abilities with it. I tried Clojure and periodically use it as a stand-in for Java, and in that sense it is good. But these days, I've been playing in Racket, and it feels like the lisp I wish I'd started with.

I'm going through a similar process. My caution is that 'package' has a very technical meaning in Common Lisp that is at odds with how 'package' is used in other languages (and a bit at odds with how the author uses it in their tutorial).

A package in Common Lisp is a set of interned symbols. In Common Lisp, systems are more in keeping with an ordinary understanding of packages...but combined with the idea of a build system just for fun.

ASDF is a way for managing systems (but it is worth keeping in mind that Common Lisp does not have any 'official' understanding of systems). ASDF is pretty much a de facto standard by consensus.

Quicklisp is a 'package manager' in the sense that it will go out and fetch a dependency from a repository. But what it fetches is a system: it is usually not a package in Common Lisp's technical sense.

From the Quicklisp FAQ:

How is Quicklisp related to ASDF?

Quicklisp has an archive of project files and metadata about project relationships. It can download a project and its dependencies. ASDF is used to actually compile and load the project and its dependencies.

ASDF is a little like make and Quicklisp is a little like a Linux package manager.

On the other hand, Common Lisp is very stable around ASDF and SLIME and QuickLisp. ASDF was started in 2002. Quicklisp in 2005. SLIME in 2003.

Not directly, but have been following various projects over the years..

My take on this is that the people using CL for machine learning have been doing this for some time, and so have their own toolsets; tensor flow is relatively new in that regard, and w/r/t lisp interfacing would entail low-level binary interfacing (and therefore mostly non-portable between implementations) to hook CL code with tensorflow kernels (definately not an expert here on either however).

also, what is popularly referred to as 'machine learning' is in my opinion mostly only one aspect of the field - e.g. classification neural networks - while lisp can definitely do this, lisp AI programming (in my amateur opinion) shines more in the realm of machine reasoning/inference due to the symbolic / dynamic nature of the environment - e.g. constructing a set of reasoning primitives (e.g. functions and facts and decision trees) and a meta-interpreter to reason/infer about external data and walk around a problem space.. also, owing to the dynamic and rapid development nature of the language, likely many people are working with their own prototype/core frameworks, possibly cobbled together from various small bits and pieces of 3rd party code. Also, neural networks have been around for quite a while - what these new frameworks bring to the table is not so much new core algorithms, but the ability to quickly cobble them together in a more popular/user friendly way, and also take advantage of fast hardware (e.g. GPUs)

as for projects - in the general sense, lisp has been a latecomer to the 'languages with cpan-style trove of public addon modules' crowd, owing in my opinion to the need to support multiple implementations in order for such a project to take hold - so older but yet still quite functional libraries might be around in various hodge-podge repositories which are not standardised but old timers have already included in their own local systems, etc. (see also CMU AI repository)

In the last few years, much has been done in the module space - I would definately consider 'quicklisp' to be roughly the defacto definitive list of current modules, especially those under active development, since the active community is basically converging on this as a module/distribution platform and so many (most?) active community projects are available as quicklisp modules and included here - and so would probably be the one of the first places to start in checking into for available libraries for any topic.

Also, the best way to 'explore' quicklisp is to install it, and then install various packages and then muck around/explore with the source code that they download into your environment - the documentation tends to be much less 'external' (e.g. websites) and much more 'internal' (e.g. READMEs, in-tree code examples or unit tests).

https://www.quicklisp.org/beta/releases.html

I don't know about being acquired but I believe there is a canadian travel-oriented startup using CL for an algorhythmically important part of their stack. I forget what it was called.

Not a startup per se, but Grammarly is a well-known company using Common Lisp for their core product.

slimv is a swank client for using Common Lisp inside vim. http://www.vim.org/scripts/script.php?script_id=2531

I used to have a problem with emacs. But just a few days ago I finally sat down to finally learn it and now, 2 days later, can no longer imagine life without it.

I've never used Emacs, so I can't really say how it compares to Slime, but I've thoroughly enjoyed using Vim-Slime[0] and Tmux[1] for development in lots of languages (CL, Racket, Clojure, Ruby, JS, SQL, Haskell, Bash, etc.).

- https://github.com/jpalardy/vim-slime

- https://tmux.github.io/

Quoting this dead (and useful) comment for others who don't browse showdead:

```

l04m33 3 hours ago [dead] [-]

I wrote a new CL plugin for Vim, which only depends on +channel and supports most SLIME features: https://github.com/l04m33/vlime It's quite new, and I'd be grateful if you could try it out and provide some feedback.

```

I wrote a new CL plugin for Vim, which only depends on +channel and supports most SLIME features: https://github.com/l04m33/vlime

It's quite new, and I'd be grateful if you could try it out and provide some feedback.

Honestly I don't think Emacs is an obstacle to people who use Vim. It's an obstacle to people who use IDEs

Really! Do you use it? Would you feel comfortable submitting a PR explaining how to set it up? I HATE recommending emacs as the IDE for Common Lisp if it's not something people are comfortable with already.

Hmm, this just gave me an idea: Visual Studio Code Common Lisp plugin might also be a good way to expose the language to people unfamiliar to the language.

It comes up in practice but is easilly avoided with the mapping of:

    null -> :null [] -> #() and false -> nil

Mapping arrays to lists instead of arrays seems wrong to me, but is what most json libraries do by default. Fortunately most of them allow you to change that.

So, I really dislike truthiness and falsiness in other languages but somehow, in Lisp, it just seems to work. One thing that helps is that, in places where the distinction between null and false matters, other features of lisp help keep them distinct: so, for example, getting a value from a hash table returns nil if the value is not found and, consequently, you can't distinguish a missing key from a stored null. GETHASH solves this by returning multiple values: the first is the value stored (if there is one) while the second indicates whether or not the key was found in the hash table. Similarly, optional arguments [declared like (defun foo (&optional bar))] default to null but, if you want to no whether a value was actually passed, you can change the argument declaration to account for this: (defun foo (&optional (bar nil bar-p))) In this case, when bar isn't passed, bar-p will b nil but when bar is passed, it will be t.

Everything should Just Work (TM) on Windows these days. SBCL provides reasonably fresh binaries to download. Quicklisp can be installed by entering the necessary commands into SBCL console. Emacs has a Windows installer. SLIME can be installed from Quicklisp. And so on. I develop in Lisp on both Windows and Linux machines, it works exactly the same (which cannot be said of some other languages).

Portacle[1] and lispstick[2] both provide portable sbcl-based development environments on windows.

Quicklisp is really simple to track what it's doing, and trivial to manage multiple installs of quicklisp simultaneously (though you can only use one install at a time in a given lisp image). Everything goes in <install-directory>/dists/<distribution-name>.

1: https://shinmera.github.io/portacle/

2: https://github.com/jasom/lispstick-automate/releases/tag/1.0

I've been telling people for years to just use Clozure CL on windows. It might be a bit slower, but it's still blazing fast and you don't have to deal with compatibility issues.

The video in question: https://www.youtube.com/watch?v=VnWVu8VVDbI&t=744s

As aidenn0 mentioned portacle will hopefully be a good option for some when finished as it gives you a complete environment in one shot.

For installing CL I may update my video to use Roswell at some point, I've been having good success with it when using TravisCI.

There are a few ways of doing it all with different tradeoffs:

If very quick startup times are a necessity, then you just build a standalone image. This is very much like deploying a program in e.g. C. You build it with asdf, and then install it (and any dynamic libraries) into your deploy environment.

For "long running" programs, this may not be as necessary, and since the lisp runtime includes a full lisp compiler, it's not uncommon to just have a script that launches your lisp executable, uses ASDF to load the system, and calls the entry point.

For either one, it can also be useful to expose a swank server, though that does have security implications (anybody who can connect to the swank server (either localhost or unix domain sockets) can run arbitrary code).

Since you mentioned Light Table, I'm going to guess that you're open to other lisps. I think this is one thing that the Racket folks get right. My wife's first foray into programming was a coursera course starting with a simplified variant of Racket, and the dev environment could not have been less of an issue.

EDIT: Also, emacs doesn't have to die. I love emacs! But DrRacket is a nice alternative for beginners (and I'm told) advanced users alike.

Lisp and Emacs have decades of shared history, it's unlikely someone will really break this bond.

It would be nice to have alternatives, of course, but there is a reason certain people like emacs. (To be fair, I was an emacs user before I came across Lisp, but I can imagine how annoying/intimidating/frustrating it can be to somebody who has never used emacs before.)

Use atom with slime: https://atom.io/packages/atom-slime

Ah, interesting! I don't remember reading that when I wrote it.

I'll go back and correct it after my current project wraps up (but PRs are also accepted!).

I am sure there are plenty of people who have a preferred Lisp. It does drive me nuts how if there is a post on R the top comments are people touting Python as being the bigger player in statistics and data science (Which it isn't) or a ton of other languages.

PS I perfer Racket :)

Nope, it has new problems that are (IMO) worse.

Sorry, there is a temporary ban on speaking about clojure in OP's thread. This discussion is about Common Lisp.

But it has many other problems.

"I wish an experienced LISPer would explain why should one use Common Lisp over a language like Golang."

Those are two very languages. Very few people would find themselves ever staring at a choice of what language to use, having narrowed it down to just those two.

Lisp is notorious for its mind-expanding freedom. I think perhaps this attribute of it is less unique than it used to be, but it is still present. While I have hard time recommending it for production usage, you can still learn a lot about programming from using it for a while. It is still one of the most programmer-empowering languages there is.

(Indeed, I think the vast bulk of the reason why it's not really all that great of a language and why it has never taken off is that it is too programmer empowering; it grants power the vast, vast bulk of programmers are not actually capable of handling well at scale. It makes it so that two programmers separated by some communication gap rapidly find it challenging to write code that works together. That said, deliberately spending some time in such an environment is an important step for the budding systems developer, I think.)

I like Go and Guile. I love Common Lisp.

It's an expressive language with a powerful development environment in the form of the REPL plus Slime. I love using Lisp for exploratory programming, as it's so simple to write something that both works now and makes sense later.

It has a wider range of built-in programming paradigms more fully realised than Go or guile. In comparison to CLOS, all other object systems are just kidding. Or you can program functionally or procedurally if you like. This gives you a better toolbox.

It's a mature and stable environment. I can run code from 15 years ago or I can update it to use the latest libraries at my option.

And I just find the regularity of the parentheses has much less of a screeching brakes effect on the experience of the code. It's smoother to read and write.

So I find common Lisp a productive pleasure to use. If I was working on a project other people had to read and maintain I'd use Go, though, as it's easier to apply skills from more common languages to more quickly.

You can run the code you wrote 15 years ago, and (hopefully) you will be able to run in 15 years the code you write today.

Stability of (most) implementations and solid language standard.

Given a reasonable choice, I would almost never use a language like Golang that doesn't enforce memory safety (at least, by default). Null pointers, ugh.