Lessons learned after working one year as a Common Lisp Developer (2018) | Hacker News Reader

Lessons learned after working one year as a Common Lisp Developer (2018)

cdagostino.io

194 pointsxept72 comments

This is "The Good" part of the lessons. There's also "The Bad" one: https://cdagostino.io/posts/2018-03-28-one-year-common-lisp-...

Great points on the list too, even though I still think they're more the acceptable trade-offs kind of bad. I.e., yes, of course `caddddr` will be unreadable, probably, so just don't. `declaim` has a sharp edge for a reason, but you can also make things better by increasing the debug and safety level, that's a great idea I really haven't seen repeated elsewhere.

The JSON library makes sense, there are several ones and they're all making different decisions wrt. representing the JSON types. I wonder if anyone made a comparison of them with https://news.ycombinator.com/item?id=20724672, that would be useful.

Lastly, I'd always add "be wary of pathnames", even though I was (am) a big fan of the concept in general, it's just sometimes a bit fiddly with regards to physical filenames that can appear on various systems (that's e.g. why SBCL's `native-namestring` can be necessary, http://www.sbcl.org/manual/#Pathnames).

> yes, of course `caddddr` will be unreadable, probably, so just don't

I've found in general, while cute that they exist, the ca*d*r family of functions are certainly a code-smell in any lisp/scheme. They're typically a sign you really want a rather different data structure than a list, and if not they can almost always be replaced with `drop`/`take`.

https://sabracrolleton.github.io/json-review.html

The differences are pretty big. Some use associated lists, some use hash tables, some convert to CLOS, etc

I'm doing this year's advent of code with literate org-mode buffers using common lisp. Been a surprising amount of fun. And I'm coming around to the idea that the loop macro is where most languages are ultimately converging.

That and format. Which is remarkable. Certainly dense, but in a "this code is clearly responsible for making a string of that list, serves no other purpose." Something lacking in basically every other language, where a depressing amount of coffee exists simply to make a string of a list of objects. (Amusing, as Java finally got streams, which can look like LOOP, but with more periods. And probably more parens...)

Many folks are doing AoC in Common Lisp and Coalton (a statically typed DSL embedded in Lisp, so you can mix and match static and dynamic code). Many are hanging out on Discord [0] if you're interested in dropping by!

[0] https://discord.gg/MCWrVfnJ

im having a lot of fun using common lisp for aoc too. for fun, my approach is to only use lists and standard lib operations on them. i am really amazed at how elegant and powerful the loop, map*, and reduce operations are. one thing i found that i needed to be mindful of are the different quality operators [0].

[0] http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node74.html

Yeah, I'm not using any libraries. Hardest part, typically, is parsing the input in. Mainly because I don't know how. Found some fun dangerous ways that work for me

I just straight used `read` for several of the days. It worked very well, even if it lacked all safety checks you might normally want.

Yeah, that is the dangerous way. :)

I actually really like it, oddly enough.

I think I used it for the bingo day, too. cl-ppcre for the first line (comma separated list of numbers) and then 5x5 loops calling read over and over to fill out the bingo cards. Why jump through parsing hoops when the language will do it for you?

I've also been solving them in Ada. It's not as convenient as just read, but if you instantiate an IO package for a type you can do the same kind of thing. So I made an enumeration and an IO package:

  type Command is (Forward, Up, Down);
  package Direction_IO is new Enumeration_IO(Direction);
  use Direction_IO;

Then reading each line was just:

  C: Command;
  D: Integer;

  Get(File, C);
  Get(File, D);

Other than the setup (3 extra lines), it's as easy as the Lisp version to use.

I also had fun for the long csv lines by making a temp readtable that set comma as whitespace and then just read the numbers in.

MAPCAR and UIOP:READ-FILE-LINES or READ-FILE-STRING will get you most of the way there.

Standard (read stream nil nil) in a loop has been working rather well. I originally was just using read-line and then for char across line. That isn't hard. Just has been a significant portion of some solutions.

Granted... The solution for some is a four line loop...

I use this:

    (defun get-file (filename)

      (with-open-file (stream filename)

        (loop for line = (read-line stream nil)

              while line

              collect (parse-integer line))))

I've so far had good mileage with basically the following:

    (with-input-from-string (stream "1,2,3")
           (let ((*readtable* (copy-readtable)))
             (set-syntax-from-char #\, #\  )
             (loop for num = (read stream nil nil)
                   while num
                   collect num)))

> literate org-mode buffers using common lisp

That sounds like a blast, I wish I'd thought of that!

i think that orgmode notebook programing generally beats anything else out of the water, including jupyter. i actually dont uderstand why it hasnt gotten bigger attention in the communities thatvuse jupyter

as for cl, if you enjoy learning it in this way you can convert norvig's paip book from md to org format using pandoc

https://github.com/norvig/paip-lisp

Org babel is great, but it's emacs only. That's why it's not used much compared to Jupyter and others. Also, there is at least one thing that would make it better: Dependency analysis across blocks for automatic re-execution. Of course, that's probably non-trivial to do across every language that org-babel supports. Pluto does this for Julia, you can break it (I have) by using certain styles in the code block (making it hard for it to detect the use) but it is nice if you work with it.

org-babel is also nice even if you aren't doing interactive programming, though. I used it to dissect a complex (150k SLOC or so) C++ program that had piss poor "documentation" (auto-generated UML diagrams and similar garbage). By converting each file to a giant source block and then extracting portions I was able to make sense of it much faster than just reading the raw code. I also left the team with my full notes which amounted to a couple volumes worth of material (code + prose) in the end.

I find it fascinating that the author was forced to learn CL because there were no jobs in Haskell. I believe him, but as a Clojure developer, I find it very hard to find a lot of jobs in any functional language, especially lisps. They exist, but I just don't see a lot of good ones. Someone tell me I'm crazy.

Common lisp is surprising not functional. It embraces that all things can be first class things. But mutable code is very very easy in common lisp.

Until everyone and their dog decided to turn functional as a synonym for as Haskell does it, Lisp was quite functional.

My understanding is that Scheme is quite functional. So were the very first versions of Lisp. Common Lisp, on the other hand, strikes me as not very functional at all (from what limited experience I have of the language). It does have first-class functions, and functions like ‘map’ and ‘reduce’, but that isn’t enough to make it truly functional — it seems to lack the characteristic approach of building functions from smaller functions that is so common in functional languages.

(Also, functionality is more of a continuum than a binary category. CL certainly is more functional than Python, but less functional than Scheme or SML. It all depends on where you draw the dividing line, though I’d argue that classifying CL as ‘non-functional’ makes for more meaningful categories.)

The very first versions of Lisp were less functional than Common Lisp: imperative, mutable data, mostly only dynamic scoping for variables, no lexical scope, first class data type of functions unclear, ...

Common Lisp at least has lexical scope and first class functions.

> The very first versions of Lisp were less functional than Common Lisp: imperative, mutable data, mostly only dynamic scoping for variables, no lexical scope, first class data type of functions unclear, ...

Hmm, perhaps I’m misremembering — I know very little about the history of Lisp. I was thinking specifically about the very beginning of the language, where it could be used to define ‘eval’ and nothing much else.

This is the manual of Lisp I, the first actual implementation of Lisp, from 1960:

http://bitsavers.org/pdf/mit/rle_lisp/LISP_I_Programmers_Man...

Common Lisp has lexical AND dynamic scoping.

This is very nice with constants. A function can make a hard dependency on standard out for example. If I change that value locally then call the function, it will operate as expected while everything else using that variable name will continue using their own expected values too. Lexical scoping would instead require passing that value in every time it is used.

Scheme directly supports call-with-current-continuation (which CL doesn't) and it requires tail call elimination (which CL doesn't require but most CL implementations do it anyway). Scheme also has slightly easier syntax for dealing with functions in function position, at the cost of potential namespace clashes.

Some schemes (e.g. Racket) provide immutable data structures.

So I guess Scheme is slightly more functional than CL but with the exception of call/cc the differences are so minor that it doesn't matter.

If functional means "higher order functions", sure Lisp is quite functional.

It's not ergonomic to compose functions in the way that Haskell is due to Lisp's forms essentially making every function able to accept a variable number of arguments. You end up with something like arnesi which provides these functions but it gets slow and verbose to write all those `funcall`'s everywhere. Or you have macros.

Whereas in Haskell, partial application and composition are first-class in the design of the language.

I really enjoyed programming in CL but I would say that it's one of those languages that let's you try functional programming but it's not the sweet spot for it.

Coalton is in Common Lisp and lets you write all that convenient syntax without much fuss.

I always understood functional in the matematical sense of: inmutable/no side effects. At some point someone decided typed lambda calculus was a superset of lambda calculus and the next thing was the haskell alike thing.

I assume you mean mutable data, and yes, mutable data is easy in CL. But at the same time immutable data is not hard; it's just not the automatic default.

Yeah, I confess I should have said "lisp is surprisingly friendly to non-functional idioms." In large, this can mean mutable data, sure.

I wish it made more things first class. There are surprisingly many restrictions.

Example: the call stack isn't first class. You can only interact with it through the condition system, which is very limited.

Example: CLOS has a meta object protocol, which is great! But there's no equivalent to the MOP for, say, the language rules themselves (first element in a list in the reader is treated as a function namespace reference, for example).

I was lamenting this the other day, thinking of Smalltalk being able to snapshot the stack and store it in a variable for debugging later. Then it occurred that one should be able to write a Lisp function that creates a data structure containing a snapshot of the whole stack and then inspect that later. So maybe no new Lisp primitives are needed?

How would you write such a function? The ways to interact with the call stack in Common Lisp are very, very limited.

The debugger of a Lisp system is also not exclusively written in the language (operators, ...) described in the Lisp standard. The standard language is largely independent from a runtime. It says nothing about the implementation technology. The Lisp system could be running on a virtual machine (like many Smalltalk systems), it could be a whole-program compiler to C, ...

For example in LispWorks it's possible to look at a snapshot of a stack in a Debugger, later.

In your last point, isn't that the evaluator, not the reader?

That said, I'm all for more first class things. I really just meant that common lisp was ahead of the game on number of first class things.

Disclaimer: Below is my experience and shouldn't be regarded as universal fact.

I've been hiring and wildly advertising Lisp jobs for around 10 years now in all sorts of domains. Good candidates always come by, but you'd think that sentiments like yours ("hard to find a lot of jobs") would mean that supply is significantly greater than demand. There is enough supply (if your employer doesn't have lots of stipulations like local-only, citizen-only, etc.; unfortunately mine does), but it's still counter-intuitive to me that supply isn't overwhelming.

I find most people who are attracted to the jobs are

1. Long-time lispers who are pretty happy with their life as it is, and would only take the job if it was truly a killer deal (e.g., work a couple hours a week at a FAANG staff-level full-time salary);

2. College students who are fascinated with programming languages and have dabbled in Lisp; and

There are others, but those dominate in numbers.

I've never, not once, got an applicant that said, "I don't know Lisp but I'd love a job with an interesting language."

Not so remarkably, Haskell programmers, Clojure programmers, and others of those ilk rarely apply. I don't know why, but Lisp has a distinctly different philosophy and lacks a solid community like those languages have.

I've sort of come to the conclusion that advertising it as a Lisp job is perhaps not even so beneficial these days. My biggest success honestly has been finding people who are obviously good at programming, showing them Lisp, then seeing them take off in less than a week and fly to new heights.

> I've never, not once, got an applicant that said, "I don't know Lisp but I'd love a job with an interesting language."

I probably wouldn’t ever apply to a role that’s in a niche language I’ve never used. Most niche roles I’ve seen though want staff/principal level domain experts though.

Last time I looked for a job in a niche language (Erlang in my case) most of what I could find was actually looking for Clojure or Scala developers or something, but decided to keyword dump every functional language they’d ever heard of.

I think the counter-intuitiveness comes from the idea that low demand must equal high supply. It could also just be an illiquid market in both directions; supply and demand both low and roughly matched, but inefficiently.

I think very much this. I've known Lisp for a longish time now (even used it at work on occasion) but never considered to seek out a "Lisp job" because language choice is far from the most important criteria in a job. At the moment I probably slot into his first category.

> I've never, not once, got an applicant that said, "I don't know Lisp but I'd love a job with an interesting language."

I don’t know Lisp and I’d love a job with an interesting language - but it would never occur to me to apply to a job requiring experience with a language that I don’t have experience in.

I think these meme job ads (we need 30 years of Go experience), as well as inexperienced technical recruiters and software engineering managers, have soured the market. If somebody is a good programmer, the last thing I care about is whether they know what NRECONC means... all that stuff can be figured out on the job.

I've not yet had a problem finding a Clojure job, if you know where to look (Clojurians Slack for example). There's not a lot of competition for roles, so getting to an interview stage is relatively easy, and because there's low competition the interview processes themselves are also relatively easy. Most positions I see, and have worked for, are for start-ups however, so maybe that's not one of the "good ones" for some people, but the pay is great, and the work interesting, for me so far at least.

So... I want to like Lisp, and every time I read stories like these I understand why people enjoy it.

But then I see a code snippet, something like this manual datetime string parser. Apparently the programmer needed to roll their own, and I wonder: how on earth am I going to convince my corpo overlords that this language belongs anywhere near a production system?

  (let ((year  (parse-integer string :start 0 :end 4))
        (month (parse-integer string :start 6 :end 7))
        (day   (parse-integer string :start 9 :end 10)))
    (- (+ (\* year   31556926) ;; no. of seconds in a year
          (\* month  2629743)  ;; no. of seconds in a month
          (\* day    86400))   ;; no. of seconds in a day
       (\* 1900 31556926))))   ;; lisp timestamps start at 1900

Pretending that something like that doesn't already exist in production, feels like a lie. As a fun example, I was looking to store durations in iso format, and saw how our analysts would have to parse that out in our data warehouse.

Well, yes, the Common Lisp standard library is showing its age in some respects. I think if you're aiming to use CL in production, you have to plan to paper over some of the standard library deficiencies with third party libraries or custom implementations. And then your concern is already accounted for.

(This goes for many other languages too. C, JavaScript, Java, Haskell, R, Perl -- they all have their "holes" in the standard library that need to be filled in.)

> how on earth am I going to convince my corpo overlords that this language belongs anywhere near a production system?

This is part of the rationale behind Clojure, being able to lean on established platforms (Java and later JS): https://clojure.org/about/rationale

With that said, though, that parsing function doesn't seem all that complicated?

The idea that “a month” has a constant number of seconds (edit: exactly 1/12th of a year) seems peculiar, so I’d argue the function needs to be more complicated.

It's using a standard-year/month/day convention which might be what's required in that particular case. It's definitely not a generic solution.

The point is that it's wrong.

Except we just brought the JVM onboard, with its own headaches. The JVM is arguably as much of an obstacle to developing solutions as boilerplate in Java. You end up having to debug obscure JVM errors in production instead of sticking to Clojure. It’s also not doing any of the compiler tricks CL and schemes are noted for.

There is no silver bullet. Clojure is a great set of trade-offs.

What obscure JVM errors do you talk about? I’ve never ever had a JVM bug and while I don’t have decades of experience, I didn’t start today either. The JVM is hands down the easiest to debug platform, where you can connect to a prod instance remotely and see basically everything. Clojure does have a few not too readable errors, but I think that’s on Clojure’s semantics, has nothing to do with the JVM.

Common Lisp has a decent (but not amazing) number of well supported libraries. It's usually not a problem, except for perhaps really enterprise-y stuff.

But Common Lisp is not the language to use in production (at least in a larger commercial context!) if your primary need is CRUD or glue. It does fine at those things, but Lisp is better wielded at problems that require "power in the large", like building a new compiler, solver, simulator, or "engine" of some sort, for instance. For those, the programming language matters a lot more than the libraries, in my opinion.

I've had no issue using Common Lisp in production and employing teams to write and maintain it.

Perhaps they didn't know about a library local-time[1]? You can't blame the programming language for the programmer shortcomings.

[1] https://common-lisp.net/project/local-time/manual.html

I once saw someone had implemented the Calendar class from Java (when Java 6 was the latest version). Apparently reading javadoc wasn't his thing.

>how on earth am I going to convince my corpo overlords that this language belongs anywhere near a production system?

Well, I don't see any difference with any other language you could use, from SQL, to C ,CSS or JSON.

The fact is that the higher your corpo overlord, the less is he going to understand your scrawl.

Instead of them understanding your language, it is you who need to speak theirs. You will need to understand MBA talk in order to communicate in simple terms(elevator pitch) why you using this weird language is important for the company.

Then they will leave you alone if what you promised is true, or just fire you if it is not.

This is true. Usually you have to convince your own colleagues, not management. (Obviously everywhere is different though.)

Your senior software engineering colleague who is a huge "Python is the second best at everything" fan will be more of a hindrance than your boss, typically. It's not wholly invalid, but it can easily become a social/political issue rather than a technical one. If someone else is proposing a popular approach (like building a backend in Python), and you're proposing an oddball one (like using Lisp), be prepared for their solution to start off with 50 points of consensus and yours zero.

Given that so much of building anything new and delivering it in a timeframe that is useful is risk management, I think that's the right place to start. Especially at a company where the risks of Python are mostly known and the risks of lisp are mostly not.

I inherited an enterprise-critical application in, I kid you not, Autohotkey. 5000 actual lines of it, none of them containing any sanity whatsoever. Apparently, some random end user decided to automate a job, other people liked, and it grew all kinds of unholy features. So after one very visible crash too many, the corporate overlords decided it was critical enough for IT to take over.

By the time I got it, most of it was calls to Win32 functions, and almost nothing was using any functionality Autohotkey was any good at. I didn't even dare talk about it on the Autohotkey fora at the time, because I was sure the developers themselves would consider me an insane idiot and/or hit me with an axe. I migrated it to another language the day I understood it well enough.

One of the great truths of ICT is: Anything can survive production if someone decides to bless it with insane amounts of time and money.

100% to that last paragraph.

One of my first jobs was converting this crazy web-first language (HTML/OS) into PHP and/or ASP.NET. The language was like BASIC for the web, and there was so much superstition based on how things actually worked (things like having an empty ELSE on an IF statement because one time something broke and that might be why). I also drastically reduced memory and CPU usage, even taking some multi-hour jobs down to seconds. Luckily this was around 2007 and I have since intentionally forgotten most of that language.

By adopting either Allegro or LispWorks, the Common Lisp environments for big corps, survivors of the Lisp Machine IDE like tooling.

Not only they are the survivors, they also come with additional libraries for all kind of stuff not covered by the standard.

Yea it’s tough. No matter how many times people say, you get the parenthesis eventually, it still is off putting for people when they see it the first time. I'm a big Lisp fan and the parenthesis don't bother me at all. But it's always going to be a sticking point with some people.

This is a big reason for the Rhombus project in Racket[1] one of it’s main goals is to be a Lisp like language which combines the extensibility of a parens based language, with infix notation and a different surface syntax

1. https://github.com/racket/rhombus-prototype

Well, let's clarify something - is it the syntax or the self-algorithm?

To compare, this is what a psuedo-popular-lang might look like:

    let year  = parse-integer(string, start=0, end=4)
    let month = parse-integer(string, start=7, end=7)
    let day   = parse-integer(string, start=9, end=10)
    
    return ((year * 31556926 +    ;; no. of seconds in a year
             month * 2629743 +    ;; no. of seconds in a month
             day * 86400) -       ;; no. of seconds in a day
            1900 * 31556926)      ;; lisp timestamps start at 1900

Honestly that looks pretty run-of-the-mill to me.

Having said that, rolling your own date logic is insanity, as the problem is difficult[1] yet solved[2].

[1] https://gist.github.com/timvisee/fcda9bbdff88d45cc9061606b4b... [2] https://common-lisp.net/project/local-time/manual.html

There's certainly a widely-used library for this (local-time). But there's also a widespread practice in CL to avoid unnecessary dependencies, partly because it's often so easy to write just the functionality you need. That's probably the case here.

This book containing interviews with Common Lisp hackers is pretty interesting. https://leanpub.com/lisphackers

I believe the author worked at grammarly, using CL. Would be nice if he wrote an updated version!

I've seen a lot posts saying condition system in common lisp is unmatched. is there similar thing in scheme?

not really. i think out of all the schemes Guile might have the most developed condition system

https://www.gnu.org/software/guile/manual/html_node/index.ht...

You can roll your own with call/cc, but I'm not aware of any out-of-the-box implementation in Schemes. Racket has an exception-handling mechanism, but it's similar to ones from other languages and it doesn't offer restarts. It does also have a more powerful notion of delimited continuations and continuation marks, which would make rolling your own condition system a bit easier (well, discounting the initial research as to what the heck they are in the first place :))

Reply on news.ycombinator.com