Show HN: Lisp Shell | Hacker News Reader

Show HN: Lisp Shell

github.com

120 pointsdexterlagan56 comments

You might be interested in an experiment of mine, https://github.com/tonyg/racket-something/blob/master/src/so..., which combines an indentation-based reader with a handler for unbound variables to permit fluid interoperation between Racket procedures and external programs.

Here's an example (https://github.com/tonyg/racket-something/blob/master/exampl...):

  #lang something/shell
  // Simple demos

  ls -la $HOME | grep "^d" | fgrep -v "."

  ls -la | wc -l | read-line |> string-split |> car |> string->number |> \
    printf "There are ~a lines here." | sed -e "s: are : seem to be :"
  (newline)

  def ps-output
    pipeline
      ps -wwwax
      preserve-header 1 {: grep "racket" }
      space-separated-columns [string->number]
      |> csv-expr->table
  print ps-output

  def message-box text:
    whiptail --title "Testing" --ok-button "OK" --msgbox text 8 50

  message-box "This is pretty cool."

`ls`, `grep`, `whiptail` and so on are unbound Racket variables, and the macros forming part of the `something/shell` dialect replace them with calls to the external programs, sorting out the plumbing, pipes and so on. `read-line`, `car`, `string-split`, etc are ordinary Racket functions from the standard libraries.

(EDIT): Here's a screencast of an interactive session with the shell. Very simple, but shows some of the basics: https://asciinema.org/a/83450

I figured out a way to combine indentation-based syntax with S-expressions too http://text.cirru.org/

Fantastic! Thanks so much for writing. I'll study all this ASAP.

Fun Unix Trivia Time: Touch is supposed to update modified times. That it creates files when they don't exist is only the most common use, not the point of the thing.

Thanks! I implemented this just to create blank files, and didn't bother researching correctness. That's exactly the kind of correction I was hoping for when posting to HN. My first post too.

Another minor issue: cd without arguments should switch to the home directory rather than display the current directory (as your Readme suggests you do). You already have pwd for displaying the current directory.

'cd' is generally a shell built-in (even with Bash) rather than a forked command so I think it's fair if shell writers want to break the mold with that particular function. It's like how the shell built-in versions of 'echo' usually differ from /bin/echo (same for 'time' as well).

So in that regard I don't think it's fair to compare 'cd' to 'cat' nor 'touch' where people will be used to the GNU / whatever coreutils.

I don't understand your argument at all. Whether something is a builtin or not is an implementation detail. 'cd' actually can't be anything but a builtin.

The behavior of 'cd' is defined by the Posix standard (in spite of it needing to be a builtin). Now, you're welcome to care or not about that. But there's no reason to make that decision inconsistently for 'cd', 'cat' and 'touch'.

> 'cd' is defined by the Posix standard (in spite of it needing to be a builtin)

I sense you're misunderstanding something, namely that POSIX defines not only the names and behavior of a number of common utilities, but also the shell language. There is no "in spite"; POSIX defines the shell syntax features like while, case, if, the various expansions that the shell does and so forth.

By your reasoning, someone's original shell-like scripting language must not have a case statement that is not terminated by esac, and that doesn't have cases terminated by ;; because POSIX requires these. I.e. one mustn't develop a program and call it a "shell" if it isn't a POSIX conforming shell.

cd is a shell feature like while or case; it's effectively a special assignment operator for mutating the shell's current working directory.

> I don't understand your argument at all.

My point being shell builtins can and often do differ from one shell to another.

> Whether something is a builtin or not is an implementation detail.

You could argue that but I think it goes a little beyond purely implementation detail. A builtin is often a builtin because it fits the shell idiom better than having a forked command. eg `time` wouldn't work well if you had to put the command you want timed in quotation marks; which is one of the reasons Bash includes it's own `time` function as a builtin despite `/bin/time` being arguably more powerful (there are other reasons but I feel this is going to be a lengthy reply and `time` vs `/bin/time` is already well documented online anyway)

Thus if this shell's idiom is simplicity over features then it makes some sense not to follow Bash's convention for `cd`. In fact if I can use myself as an example; I have also written my own shell I didn't follow Bash's convention for `cd` either (which means I'm also aware `cd` cannot be anything other than a built in :P) The reason I broke compatibility with `cd` was because I wanted historic directories in a stacked array by default; I don't support `pushd` / `popd` because `cd` does it right out of the box. The historic variable holds the previous directories in a JSON array and that better fits the idioms of my shell because my shell is designed around natively munging data structures such as JSON. Thus my builtins should also follow the same idioms as the rest of the shell regardless of whether that breaks POSIX compliances or not.

> The behavior of 'cd' is defined by the Posix standard (in spite of it needing to be a builtin).

In case you hadn't noticed, this isn't a POSIX shell. There's no law stating every $SHELL has to be POSIX compliant and in fact a lot are not; including Bash itself. So your point about POSIX compliance is not relevant here at all as if you want a fully POSIX compliant shell then you should be looking elsewhere.

> Now, you're welcome to care or not about that. But there's no reason to make that decision inconsistently for 'cd', 'cat' and 'touch'.

Again I feel the need to reiterate the point that one runs a different shell because they lean towards a different idiom. So with that regard there is every reason to make a decision for breaking standards on `cd` if those standards don't conform to your shells own idioms. Otherwise what the hell is the point of running an alternative shell?

I feel `cat` and `touch` are exceptions to this rule because they are traditionally forked. I mean by all means the author is welcome to re-implement some of coreutils as well if they wish (ala Busybox) but I personally feel that is a step too far in shell design (excluding for a moment my previous example of Busybox as that is clearly intended as a one stop shop and is very useful in that regard!). I think the saner approach if you want to offer alternative functions to coreutils is to offer alternative builtins (eg `lcat` for "Lisp cat"). However that is my personal preference and if this particular shells author wants to reimplement coreutils for whatever reason they choose then I wish them luck. If nothing else, it will teach them a new appreciation for GNU et al coreutils.

There's no law stating every $SHELL has to be POSIX compliant... one runs a different shell because they lean towards a different idiom.

I absolutely encourage that! I accepted that possibility in my comment. See the kind of stuff I build: https://github.com/akkartik/mu. I introduced Posix purely as an example that builtins are as subject to standardization as coreutils.

Other than clarifying that I'll agree to disagree. It's great to rethink a shell from the ground up. If you're doing that, what earlier shells happen to choose as a builtin is an irrelevant signal. Depending on how you design your idioms, it may make sense to override stuff from coreutils as builtins. As you've pointed out, such overriding has already happened in bash and other shells.

These are all artificial boundaries to be questioned. Decide what behavior you want to copy and what you want to rethink regardless of where you find it.

Sorry but now it's my turn to not understand the point raised. Are you arguing that people should be allowed to explore other options outside of POSIX compliance but except in this specific case?

It's pretty clear this shell isn't intended to follow in the footsteps of Bash, let alone be fully POSIX compliant, so I don't really understand why you are comparing it to them in terms of compatibility (and even more confused now that you've said you encourage people breaking POSIX).

Or is the point you're raising a question of what should be a builtin and what should not?

(happy to agree to disagree by the way; but I'm just a little confused as to what we are disagreeing about :))

Best kinds of conversations :)

Set aside my mention of POSIX as a minor point. My basic claim is this: building a whole new shell is an ambitious act. Whether your goal is to create a better experience for others or just learn new skills, restricting yourself to what earlier shells happen to consider to be builtins is limiting ambition. Be more ambitious and rethink as much as you want.

I'm not saying you have to rethink everything all at once all the way up from machine code (though I'm sympathetic to and experienced in that particular failure mode ^_^). I'm saying if you have an idea to improve 'cat' to fit better with your new shell, the fact that it's in coreutils shouldn't cause you pause. Who cares if it's a builtin in bash or not? Make it a builtin in your shell.

Ahh in that case I completely agree with you :)

Unless you’re seven directories deep and accidentally hit the <enter> button instead of the <shift> button.

In which case, uh, you want the current functionality.

You can use "cd -" to go back to the previous directory.

$OLDPWD also holds the previous directory.

If you use "mkdir -", bash will not change into this directory. Not even if you use "cd -- -". "cd ./-" works.

Does that work like a stack or does it just swap?

It only swaps; you can use pushd and popd if you want a stack.

Also:

- "cd -" has no interaction with the pushd/popd stack except in the sense that it alters the top of the stack (the current working dir). It doesn't swap the two top entries of the directory stack, but swaps the top with an off-stack saved location ($OLDPWD).

- "pushd" with no arguments swaps the top two stack entries.

To create a blank file, another idiom is "> foo".

I suppose this is a similar situation as with `cat' - it's meant to conCATenate stuff, but the most common usage I've seen (and one I've learned) is the idiom of `cat somefile | some-commands'.

(Whenever I share a snippet with this idiom, I get told by one of my cow-orkers to stop abusing kittens...)

Isn't `some-command < somefile` more idiomatic (also, should spawn one process less)?

Yes. See "Useless use of cat" (or "UUOC").

If you prefer to have the command arguments at the end of the line (so it's easier to amend), you can do e.g:

  < file.txt grep pattern

instead of

  grep pattern file.txt

  cat file.txt | grep pattern

There are also some related useless uses, like useless use of grep | awk:

  grep pattern file | awk '{print $3}'

which can instead be written as

  awk '/pattern/ {print $3}' file

Oh, the update times feature is pretty common, at least among those who use a shared system where "scratch" files are deleted when they are seven days old.

Funny thing. I work in the computational field where we deal with TBs of data regularly. We are supposed to backup sim outputs to tape asap and using touch to modify modification times to avoid this is explicitly forbidden.

Weird. I learned these two things in the opposite order. I guess my first use of touch was to test out makefiles

This reminds me of my time working with the Scheme Shell scsh!

From the readme I don’t quite understand the purpose of lsh, though. Usually, a shell allows you to run programs and manipulate their environment. However, lsh seems to reimplement some commands like find or rm and does not provide operators to manipulate the runtime environment like redirecting output. Maybe you can write in the readme your goal: play with racket, have a shell that accepts racket forms or something different.

It would be nice to see some examples of the syntax, maybe a couple of simple scripts...

E.g., the README states that 'cd/' is an alias for '(cd "/")' which seems to imply that one has to wrap every command in '()' and quote every filename... not very practical for a shell!

Thanks for the feedback, I will update the description ASAP. I know I need to write a few examples, especially how to handle files. If compiled with pmap, it's possible to do things like:

(pmap (lambda (f) (run (+ "[some-command] " f))) (directory-list))

...and run any binary against all files in the current directory, in parallel. (directory-list) returns a list of files and one can run any Racket function against that list. It's also possible to gather any command output and pipe it back to Racket for processing. I'm working on a more polished build with better docs.

  I'd like to cover the minimum (like running binaries directly instead of having to use run). I've only begun to think about everything that's needed to bring this to its full potential.

  I'll also check the other projects I hadn't heard of, maybe some of them are already far better than mine :)

Thank you for writing. Agreed, there a need for direction and better documentation. Feedback has been great and this has motivated me to work harder at this. I’d like to have something more standard. I currently use lsh to walk paths and record macros I use in IO tools for production.

For the Emacs users, there's eshell - a Shell, implemented in Elisp with the common GNU tooling, but also with the option to hook into regular elisp functions. Pretty cool stuff!

With the major caveat that piping commands together doesn't work that well, eshell has been an amazingly useful feature of emacs for me. Combined with TRAMP support, it makes dealing with remote machines much much more friendly. As an example, "cd 'ssh:firstmachine|ssh:secondmachine:/'" works just like you would anticipate, bounces you through first machine to second machine. All commands from that point are executed as if you were on second machine. You can even run "shell" to drop to a more traditional shell at that point. Even better, "cp someFile ~/" will transmit a file back to my local machine so that I don't have to work with the scp commands that it requires. :)

> Even better, "cp someFile ~/" will transmit a file back to my local machine so that I don't have to work with the scp commands that it requires.

Did not know that. That’s a neat trick! Do feel free to share more if you have any :)

Some time ago, I wrote a small blog post about that Tramp and remote editing files with eshell: http://200ok.ch/posts/edit-remote-files-with-emacs.html

Don't really have much else. I do have "M-x g" bound to "magit-status" so that as long as my PWD is in a repository, no matter if it is remote or not, I get my git status buffer. That didn't really require any setup, though. Magit already "does the right thing" if you are on a remote machine.

I did have to add a lot to my tramp path. Can't remember why, right off. I just know that it helps find a lot of things that don't make the standard path for some reason.

What I love about it is that elisp functions can either be called like lisp functions or as unix-style commands. For example:

$ (defun sq (x) (* x x))

$ sq 5

I want to like Eshell, but it's hard to get past the fact that it has existed for almost 20 years and still doesn't support basic input redirection. Modern shells like Fish and Zsh are far more polished and convenient command-lines.

I really like seeing more people use Lisp, and it's neat to see what other people come up with, but I'll stick with Emacs, Slime and occasionally eshell.

More often than not it's easiest to just use the Common Lisp functions for creating directories and interacting with the system, but when that doesn't work, I have a function similar to this one in my .sbclrc file:

(defun run (cmd) (with-output-to-string (outs) (uiop:run-program cmd :output outs)))

At work, I've even created a small Common Lisp library for calling our JSON APIs over HTTPS and running commands on our test clusters using SSH. It's all tightly integrated into Emacs, and I can use it do things like open remote files in my local Emacs.

RASH, RAcket SHell library and language: https://github.com/willghatch/racket-rash

Note: I am not the author

I started lsh a few weeks before finding out about Rash. I found it too complicated to be used as a replacement for sh, but I might not have read enough about it to make a proper judgment. In the end I wanted to implement something that behaved exactly like I'd expect to.

I'm late to the party, but I'm the author of Rash. Rash's documentation is terrible and out of date, but I've actually been planning to redo/improve all of the documentation within a week or two. I'm hoping that once I do that it will be much more approachable. Rash tries to allow both Bash style things and more normal Racket to be done with ease and mixed, and a lot of the complexity of Rash has to do with getting those two things to work together, and to allow lots of macro extensibility. But I think you'll find that it's quite powerful and useful... once I make it easier to make sense of.

Of course I'm also happy for anyone to make a new shell that they like and want to use themselves. I did it, after all...

Hi there, thanks for replying. I’m quite sure Rash is superior to my toy program, tbh I can’t wait to read more about it and check your code. If I have some free time I’ll try to contribute too. Cheers!

I was going to comment about this! The author is a student at my school who studies under Matthew Flatt (so no surprise it's based on Racket). I've worked with him on class projects before and he's not only wicked smart but also a really nice guy.

CLASH: CLisp As SHell http://clisp.org/clash.html

I've never used it as a shell per se, but another little tweak I do sometimes for shell script type things in Lisp is install a read macro like

    (set-macro-character
     #\$ #'(lambda (stream char)
             (declare (ignore char))
             `(uiop:getenv ,(symbol-name (read stream)))))

This lets you read environment variables like $home, and write to them like ordinary variables, eg (setf $message "hello"). It reads the variable name as a Lisp symbol, so by default it will be upcased; this happens to be convenient for environment variables, but as with any symbols in Lisp, you can escape lowercase characters using either \ before the character or |'s around the whole symbol, or you could change the readtable case, or just preserve case initially in the read macro.

suprised no one has posted the venerable https://en.wikipedia.org/wiki/Scsh

The most memorable thing about Scsh for me is its acknowledgments rant, which starts:

"Who should I thank? My so-called "colleagues," who laugh at me behind my back, all the while becoming famous on my work? My worthless graduate students, whose computer skills appear to be limited to downloading bitmaps off of netnews? My parents, who are still waiting for me to quit "fooling around with computers," go to med school, and become a radiologist? My department chairman, a manager who gives one new insight into and sympathy for disgruntled postal workers?"

and concludes:

"Oh yes, the acknowledgements. I think not. I did it. I did it all, by myself."

http://philip.greenspun.com/wtr/dead-trees/acknowledgments.h...

Scsh home page here: https://scsh.net/

See also clojure shell https://github.com/dundalek/closh

Interesting but one nitpick: always, always, always, in the github readme.md put an example of use so I can see what it’s like before installing.

Agreed, I will update the readme ASAP with examples. This is my first post ever, and I didn’t expect such interest. Thanks a lot for writing.

Kudos for being brave to share. Keep up the good work and thanks for being open to input.

As long as the top-level commands don't have to be manually wrapped in (), I like this idea.

Thanks for writing. That's why I made this little shell in the first place: I wanted something quick that worked mostly like sh, but with a Racket inside.

Reply on news.ycombinator.com