Scripting in Common Lisp | Hacker News Reader

Scripting in Common Lisp

ebzzry.io

130 pointsmr_tyzic38 comments

It's not common lisp, but most of my Emacs use has been as a shell scripting alternative.

Workflow being based around interactive improvement of a script from a simple interactive-macro through to things that are effectively done as shell scripts.

The end result is something that can be run as a shell script, although translating to another language i.e. CL, Ruby, Go, etc. is the usual path for me to make better use / performance.

The old joke about Emacs being an operating system that needs a good editor is on need of a reboot.

Emacs is an interactive computing and development environment with half a dozen good text editors.

Factual isn't as funny though.

Emacs is my favourite software, but elisp is one of my least favourite lisps. The other day I stumbled upon Shen language after some years and noticed it now has BSD license and an elisp implementation. I tried to add it to spacemacs and, to my surprise, it just worked. When I have some spare time the next thing I'll do will be to learn some Shen and see if for shell scripting it is an improvement over elisp.

Wow! thanks for pointing out there is an Elisp Shen implementation!

Erlang too, fwiw.

I'm somewhat baffled that Hy didn't gain any traction.

Let's review the facts:

- A lot of people love Python.

- Plenty of people love Lisps, afaict with a sizeable overlap with the first group.

- Hy, which is a Lisp on the Python platform, has no popularity whatsoever compared to other Lisps like Clojure.

The heck?

Hy may have changed since I last looked at it, but when I had looked at it, Hy was more of an s-expression syntax bolted onto python.

Some random thoughts on why this matters:

Clojure is very much not a thin s-expression layer for java; that would have been terrible (Python is more like lisp than java which is why Hy is tenable whatsoever).

In particular, CL style macros with out some first-class concept of symbols are a footgun[1], but I'm sure I could find other issues.

Another item that can't be ignored is that Hy selected the clojure syntax, which is going to be a turn-off for anyone coming from Lisp or Scheme.

Finally, there are hundreds of thin s-expression layers on top of existing languages, and most of them are terrible; Hy has done nothing to stand out from this crowd, so most lispers will not look closely at it.

1: See https://youtu.be/cPNkH-7PRTk?t=4786 for Rich's discussion of how he managed this issue in Clojure.

I'm actually interested in what other problems make Hy and similar languages infeasible. Even if a brief overview, as food for thought on the matter.

They aren't infeasible; a real lisp could target Python. However, just bolting on lisp syntax to python ends up in an uncanny valley of "not python" and also "not lisp."

I would actually recommend watching the entire youtube video I posted earlier (Clojure for Lisp programmers). Rich talks about a lot of the design decisions that went into Clojure. Some decisions I agree with, some I don't, but there was thought into creating a unified whole and he was clearly aware of work that had gone before, even when he chose to do things differently.

Note also that Clojure was his third attempt at doing a lisp/java interop, so there was a lot of previous experience backing many of those decisions.

Lastly, it's really easy to make a bad lisp-frontend for a language and a massive amount of work to make a good one. This is well known by anyone who has been in the lisp community for a while (if for no other reason than they themselves have made a bad one) so for better or worse, such frontends tend to be ignored.

A good way to target a Lisp to Python might be to forget the Python AST and target its virtual machine directly.

The compiler could itself be written in its own target dialect, and bootstrapped with some existing Lisp.

If I had to guess I would say the overlap is not as sizeable as you think, at least in the Common Lisp and Scheme camps.

Python really has very few things in common with Lisp and its philosophy is in many ways orthogonal to the philosophy of Lisp. From the idiotic moto "There should be one — and preferably only one — obvious way to do it" (which isn't even true in Python-land) to all the compromises that Python accepted to attract marketshare that would make any hardened Lisp user want to barf, it's crystal clear to me that Python makes an even worse platform than the JVM for Lisp.

Some examples of serious differences between Python and Common Lisp - that would make any serious CL user run away from Python/Hy - are:

+ CL (like Smalltalk) is a fully interactive language. Python's focus on interactivity is minimal if not entirely unproductive. Approaches like Jupyter come across as terrible hacks that lack most of the features that matter to people immersed in Lisp.

+ CL can be very performant. Implementations like SBCL can produce native code competitive with C compilers.

+ CL is truly multiparadigm (one can do imperative, functional, OOP, logic, aspect, nondeterministic, lazy, parallel/multicore and not feel as if the language is fighting him every step of the way). Python has too many limitations for that to be true.

+ Lisp feels conceptually clean and internally consistent. The metacircular evaluator is a thing of beauty. Python feels like a collection of gross hacks and adhoc rules thrown together by humans who didn't know better.

For me it is language instability. I was trying with Hy multiple times, but every new release they would break something, mostly by changing or removing core functionality (eg. removing let/cons).

To me, this looks like kids playground rather than serious project.

If Hy developers doesn't know how to evolve language properly, especially hosted one, which is challenging for itself, I'd suggest they look at Clojure or take some proper language development course.

It doesn't surprise me. It does not have a strong selling point.

I go on to the documentation page, and see that yes, it is nice clean lisp-like language, but I can't see what problem it solves that Clojure doesn't already solve. (And clojure has the existing library/community support to give it the edge.)

EDIT: That's not to say that there isn't one. I just am not immediately seeing it.

> which is a Lisp on the Python platform

'a Lisp' is a practically useless term, when you expect that usual Lisp code will run or even being able to be ported. It's incompatible with any other Lisp. Which makes it an island.

Played around with it a few months ago. The syntax is peculiar, clojure-inspired, but still very different. Breaking changes have been made, leaving most libraries unusable. The documentation didn't mention this (and is lacking in many other ways).

And btw. many of these changes weren't even features. Like core functions getting removed for philosophical reasons.

Feels like they broke and abandoned it

I imagine Clojure is more popular with Java programmers than it is with CL programmers. I did CL for over a decade and I've certainly never had a need to pick it up:

Java programmers on the other hand always seem to be looking for a reason to not use Java. But Python programmers? Always looking for a reason to use more Python.

FWIW: A lot of people also don't love Python. I particularly hate it, but the things (I perceive) it's bad at, Hy doesn't help with.

>- Plenty of people love Lisps, afaict with a sizeable overlap with the first group.

Not that many. And not that many are non-pragmatic enough to mess with a translation layer.

They removed "cons" in the latest version :(

Python can't use multiple cores, in contrast to the other lisps?

What does language have to do with "use multiple cores"? Python has both multiprocessing and threading libraries.

And the GIL on CPython, but yeah multiple process are also an option.

This is the first time I've seen content provided in Esperanto like this. Pretty cool.

Even cooler is that the Esperanto version seems to be the original, and then I suppose it was translated to English a year later.

Huh, now I see the whole blog defaults to Esperanto.

I wonder how much of the net takes that stand of Esperanto-first.

I use scripting extensively in Clojure, both on the JVM and on Planck (JS). You have the choice of slower startup times but a ton of libraries to do anything you want, versus short start-up times but less libraries, using the same language. I use https://github.com/l3nz/cli-matic as a wrapper and it works on both.

Using modern Clojure's clj is so much easier than the CL procedure.

Until you have to deal with unintelligible Java stack traces.

They are intelligible if you know Java and Clojure 1.10 improved stack traces.

Yup.

Common Lisp stack traces tend to be no better, anyway, at least with SBCL and CCL. Up to 90% of a trace you see in the inspector can be compiler/runtime infrastructure. Given that macros don't show up, only their expansions, the backtrace can be full of stuff like SB-KERNEL:%FUNCTION-THAT-I-DONT-RECOGNIZE-BUT-MACROEXPANDED-FROM-SOMETHING-I-KNOW.

Like in Java, most of the time the top few lines are the ones related to your code, but just like in Java, there are exceptions, in which the source of the problem ends up in the middle of the backtrace, surrounded by infrastructure frames.

This is a great guide, and solving quick real problems is how people get started programming in general.

I think seeing scsh for the first time made me realize just how good & bad shells are.

> I think seeing scsh for the first time made me realize just how good & bad shells are.

Oddly enough, just last week I was trying to do some scsh-like things with Common Lisp, and I ended up banging my ahead against its case-mapping behaviour (Common Lisp upcases input by default).

There’s definitely an opening for a Lisp version of something like scsh. There’s some tricky stuff to be aware of (e.g. SBCL’s RUN-PROGRAM puts processes in a different process group, which mean C-c does the wrong thing), but done well it could really be awesome.

Setting case to invert by default fixes a lot of those problems (you just invert the case again before passing the string onto the shell).

Also, for doing many shell things fork/execv is the only thing that gives you sufficient control, so I suggest avoiding run-program and using those instead.

I recommend roswell https://github.com/roswell/roswell .

Pretty nice considering the speed and capabilities of [SB]CL relative to other mainstream scripting languages.

I shy away from CL scripting due to runtime startup time, though if you compile it down to binary it's manageable (if one doesn't mind a 50MB executable).

That said, I wonder why nobody tried a "hybrid" solution, in which the multi-call binary is implemented as a resident, self-restarting process, that executes each request in a lightweight sandbox. Actual shell commands would then be implemented as a request to that process, and incur only the costs of setting up IO streams on startup.

what CL implementation ? sbcl ? I don't know what's the leanest CL out there (even if not the most featured, since we're talking about short scripts)

SBCL mostly. I probably should give GCL a try here. SBCL is fast as hell once it starts up, and bare SBCL starts up quickly, but once you add Quicklisp and start loading external systems, the startup time gets noticeable - though again, this gets reduced quickly iff you dump a binary of your image with dependencies already loaded. The one CL script I wrote that I actually use[0] (for controlling Hue lamps) I have dumped into binary just to cut down startup time to below the point I notice.

[0] - https://github.com/TeMPOraL/hju/

SBCL does have a tiny delay (akin to a raw JVM). CLISP is near interactive though. But that's without quicklisp.

Where is your piping example?

I do not use cl-launch, but for some tasks at job I wanted to pipe programs, and just wanted to try implementing something myself. The with-pipeline macro creates temporary fifos to connect programs and threads (random example):

    (let ((counter 0))
      (with-pipeline ()
        (program "ls" "-1la")
        (lambda-line (line)
          (format t "~x~%" (incf counter (length line))))
        (program "sed" "-n" "s/A/_/g ; /__/ p"))
      counter)

Outputs and return:

    2__8
    15399

Macroexpansion:

   (catch :pipeline
     (block nil
       (let* ((#:input%2195 (pipeline::ensure-stream nil :input nil))
              (#:output%2196 (pipeline::ensure-stream t :output nil))
              (#:error%2197
               (pipeline::ensure-stream :output :error #:output%2196)))
         (let ((#:g2199 (program "ls" "-1la"))
               (#:g2200
                (lambda-line (line)
                  (format t "~x~%" (incf counter (length line)))))
               (#:g2201 (program "sed" "-n" "s/a/_/g ; /__/ p")))
           (pipeline.pipes:with-pipes% (#:pipes2198 2)
             (lastcar
              (mapcar #'pipeline.filters:clean
                      (list
                       (pipeline.filters:spawn #:g2199 :input #:input%2195 :output
                                               (pipeline.pipes:pipe-out
                                                (svref #:pipes2198 0))
                                               :error #:error%2197 :wait nil)
                       (pipeline.filters:spawn #:g2200 :input
                                               (pipeline.pipes:pipe-in
                                                (svref #:pipes2198 0))
                                               :output
                                               (pipeline.pipes:pipe-out
                                                (svref #:pipes2198 1))
                                               :error #:error%2197 :wait nil)
                       (pipeline.filters:spawn #:g2201 :input
                                               (pipeline.pipes:pipe-in
                                                (svref #:pipes2198 1))
                                               :output #:output%2196 :error
                                               #:error%2197 :wait t)))))))))

https://github.com/christophejunke/pipeline

Here's an example from my `pelo` project:

https://github.com/zhaqenl/pelo/blob/master/pelo.lisp#L112

  (defun get-ping (host)
    "Get ping reply from host."
    (inferior-shell:run/ss
     `(inferior-shell:pipe (ping -c 1 ,host) (grep "time=")
                           (sed -e "s/^.*time=//;s/ 
  *ms$//"))))

The idea is from a similar project in a different Lisp dialect:

https://github.com/ebzzry/pell/blob/master/pell#L124

Reply on news.ycombinator.com