Shell Has a Forth-Like Quality
oilshell.org
oilshell.org
I also like that the author mentioned runit. I wish there was more documentation for runit, but I've been using it on Void linux and it's an amazing, damn simply init system. If an init script is missing for any package in Void, they are trivial to write. Most are less than three lines.
I also really like how the post author demonstrated how shell commands are composed in ways that traditional functions aren't composed; that is commands that change the environment for the command they run (environment variables, processor scheduling, security, input/output redirection). This is the very nature of an _implicit_.
What does it add exactly? As far as I can tell, it's needlessly incompatible with POSIX shells, and doesn't do as much as e.g. zsh (or even bash?).
I'm not saying that zsh & bash are the end-all, be-all of shells, but I just don't see that fish improves on them.
And honestly, for config files I think we could do a lot worse than an s-expression language.
* Syntax highlighting
* Implicit cd to directory
* 100,000 line history
* History de-duplication
* Inline auto-suggestions
* Tab completion using man page data
* Paginated completion
* Copy/paste using X clipboard
The only things I feel the need to configure on a new fish install, is PATH and my prompt. Because I'm pedantic and like things exactly so.
You can get these with zsh, but it's more than apt-get install.
I do not want that my lines get deleted silently. ever. I would rather prefer that random files on my disk are deleted, before that. Is there any shell without line history limit?
bash's history limit is by default the last 500 commands, without considering duplicates. In practice this limit is easy to exceed, which is why bash config guides recommend upping it.
fish's history limit of 2^18 (~256k) commands is really more of a sanity check. Duplicate commands are counted only once, and commands are discarded by LRU (not oldest). This means history only starts being dropped after hundreds of thousands of unique commands. If you in fact run that many distinct commands, I'd love to hear more about your use case!
export HISTSIZE=-1
export HISTFILESIZE=-1
Here's where I think fish fits in. The main goals of fish are:
1. Thoughtful UI that just works
2. Sane scripting
zsh does more than fish, and we're happy to let it. The tradeoff is that zsh requires substantial configuration to realize that power - and once you get into the weeds, you will learn which options are incompatible with each other. zsh has 21 options controlling history, and fish has none, but you'll probably be happier with fish's defaults.
fish has a smaller feature set, but that feature set just works out of the box: tab completions, syntax highlighting, autosuggestions, etc. Customizations (prompt, colors, etc) can be done with a web UI or a nice command line interface, and not arcane syntax (PS1=...)
The hope is you can be more productive in your shell without needing to invest tons of time configuring and learning it.
Parent's comment is misleading, then. As someone who already has a pretty solid grasp of Bash, when I peeked at Fish it looked like there was nothing that it would gain me -- though I understand that the preconfigured shell can be useful for people who don't have the time to learn and tweak it.
If you're a die-hard bash user, autosuggestions are the feature most likely to pique your interest and make the command line more pleasant.
The biggest amazing default is to be able to start typing and have it be searching the history. When I'm ready to start work on a project I can start typing part of the project's name, say "bar", and fish will come up with a history item for "cd /customers/foocorp/barproject".
I still do my "scripting" shell work in bash though, since it runs everywhere.
Probably (although I use a zsh distribution to get a sane setup). I definitely appreciate the idea of sane defaults, rather than forcing the user to set everything up properly from scratch. And your autosuggestions do look pretty swell.
> Customizations (prompt, colors, etc) can be done with a web UI or a nice command line interface, and not arcane syntax (PS1=...)
You ship a web server in the shell? Is it always running, or does the user have to do something to start it up?
I kinda like setting variables, to be honest. But I'm old-fashioned.
One thing I really wish y'all'd done was use more standard shell syntax. It's really painful working in an office where someone uses fish and lets fish-isms creep into shared command libraries.
> Is it always running, or does the user have to do something to start it up?
Disclaimer: I am the author of elvish.
https://github.com/oilshell/oil/wiki/ExternalResources
If there is a better summary, let me know, or feel free to edit the wiki page yourself. (I think it should be editable.)
I've been in contact with the authors of the oh shell, NGS shell, and mash shell, and exchanged some interesting ideas. The design of the oil language should be more concrete in 2-3 months and I'm hoping to get feedback from others who have thought about this space. I might ping you then :)
Are the elvish structured data pipelines actually separate processes, or are they goroutines? Is there a format for piping them over say SSH? I have been thinking about this a little bit for oil.
Structured data pipelines run in the same process as elvish, in goroutines. For interacting with external commands you need to serialize and deserialize, like this:
~> put [foo bar lorem ipsum] | to-json |
python3 -c 'import sys, json;print(json.dumps([x.upper() for x in json.load(sys.stdin)]))' |
from-json
▶ [FOO BAR LOREM IPSUM]I had some good discussions with Michael MacInnis and recently Ilya Sher. I had actually read Michael's masters thesis in 2011 or 2012 and found it to be a very good historical overview of shells:
https://scholar.google.com/scholar?cluster=99934401164441475...
I agree with the goal of "taking shell seriously as a programming language", which I think all projects share. I disagreed on the need to make shell a variant of Lisp. I actually tried this a bit -- I hacked on femtolisp, which is used to bootstrap Julia, and I thought I could bootstrap a good shell with it. This didn't work out as nicely as I had hoped so I dropped the idea.
Ilya was having some issues with the "two modes" of parsing -- command mode for shell-like constructs and expression mode for programming features (function calls, math, etc.). I thought this would be an issue too, but as mentioned in another comment, now that I've written a bash parser I think it's straightforward to solve, and Ilya agreed.
The thing is that bash already has many different modes of parsing, which I implement using "lexer modes" or lexical state. This same technique can be applied to a new shell and then there should be a natural way of mixing commands and expressions (quoted and unquoted words, etc.):
http://www.oilshell.org/blog/2016/10/19.html
I'm also trying to make oil do what bash/dash/etc. do essentially syscall for syscall. I believe that is a little hard with Go because it has its own runtime. Although I think there is a way to pin a goroutine to a single thread which can mitigate those issues.
I was planning to start a thread in maybe a month to review some of these issues and exchange feedback. Let me know if you are interested!
Reminds me, I have a pretty neat tail-recursive shell pipeline trick I need to share with HN!
The next post mentions that pipelines are also concatenative, although there is a slight difference and I think "point-free" is more accurate.
"Pipelines Support Vectorized, Point-Free, and Imperative Style"
For instance, one could write "is_installed(X11) :- is_installed (KDE)" to specify that X11 must be installed if KDE is installed.
http://www.redditmirror.cc/cache/websites/web.archive.org_84...
(actually, parsing is one of the things i really love about Prolog. Writing a simple grammars is much easier and much more pleasurable than using things like regular expressions. The fact that it's far more powerful and readable is just a bonus. )
(edit: incidentally, I've also had the same thought about CSS selectors.)
It's functional and not logic though:
But Nix is certainly an improvement over traditional, imperative package management.
Lots of the little DSLs used are in fact the descriptive abstractions you mention.
If not, it's quite easy to implement Prolog or any other logic language like Kanren in Scheme. It's been done many times.
As a thought experiment:
- Let distros, package maintainers, repository maintainers and the guy building define boolean flags (like "UNIX", "Linux", "ARM", "ARMv7h", "x86", "x86_64", "POSIX_Filesystem", "GCC", "Clang", "C++11", "C++14"... anything you could think of). The latter override earlier definitions (in the order above, i.e. build > repo > distro). Allow the deriving of flags for simplification: "ARMv7h -> ARM", "Windows && ARM -> Use_some_WinRT_Hack" etc.
- Use a Prolog-Like system in which you can state rules for internal flags deciding if/which files should be included. Same goes for compiler flags.
- Add an option to add definitions for external projects that can be included automatically from their sources/binaries (using the same system; like Meson does).
- Just implement the "download/bootstrap" part and the logic solver. Generate ninja or tup files.
That should make an interesting build system. Of course, the maintaining of source file dependencies (which can be dynamic due to header files) and the versioning of libraries would still be problems that are not easy to solve. But at least, one could get rid of the mess that is pretty much every CMake build there is.
Turing-complete package metadata has some obvious disadvantages too.
http://skarnet.org/software/execline/pipeline.html
Execline resembles a "program launcher".
We can make a braindead launcher in a few minutes using the sh language. This is just the first way that came to mind.
#!/bin/sh
test $# -ge 1||
exec echo usage: $0 program arguments
test $# -le 5||
exec echo max 5 arguments
{
a=$#;
b=$(command -v $1);
echo 'main(){
char *b['$((a+1))'];'
a=$((a-1));
test $a -ge 0||
exec echo 'b['$#']=(void *)0;
execve("'$b'",b,(void *)0);
}'
echo 'b[0]="'$1'";'
# repeat
a=$((a-1));
test $a -ge 0||
exec echo 'b['$#']=(void *)0;
execve("'$b'",b,(void *)0);
}'
echo 'b[1]="'$2'";'
a=$((a-1));
test $a -ge 0||
exec echo 'b['$#']=(void *)0;
execve("'$b'",b,(void *)0);
}'
echo 'b[2]="'$3'";'
a=$((a-1));
test $a -ge 0||
exec echo 'b['$#']=(void *)0;
execve("'$b'",b,(void *)0);
}'
echo 'b[3]="'$4'";'
a=$((a-1));
test $a -ge 0||
exec echo 'b['$#']=(void *)0;
execve("'$b'",b,(void *)0);
}'
echo 'b[4]="'$5'";'
a=$((a-1));
test $a -ge 0||
exec echo 'b['$#']=(void *)0;
execve("'$b'",b,(void *)0);
}'
echo 'b[5]="'$6'";'
} #| exec gcc -xc - -o run -static \
&& exec run
"no stinkin loops"
or "stupid shell scripter" PoC, take 2
#!/bin/sh
# e is a dir of "environmental variables"
# e is mounted on tmpfs i.e., it's memory not disk
# output:
# ==> 0
# ==> 1
# ==> 2
# ==> 3
# ==> 4
# ==> 5
test -f e/$0||echo 0 > e/$0
read n < e/$0
test $n -le 5||exec rm e/$0
echo ==\> $n
echo $((n+1)) > e/$0
test -f $0||exit # just in case
$0Why are terminals NOT just REPLs for regular programming languages with OS_Convenience_Functions #include'd automatically?
This all cascades to make a language that looks much different than you would want to write a full application in. One of the primary reasons is because of how shells need to handle strings. It is much more convenient to not type out quotes around string literals and it is nice to have syntax to reference environment variables, files, previous commands, etc. This leads to extremely complex quoting and expansion rules in most shells. It is not intuitive, but once learned it makes for a great interactive experience (compared with traditional languages).
Shells have also been built to help glue applications together, and are more concise for the easy versions than most programming languages, sometimes heavily so. Most programming languages, (outside of maybe perl and ruby), don't make this use case anywhere close to their top priority, but shells do, trading off other things like a simple syntax for bitwise OR.
In another sense, shells are exactly what you describe, it just happens that the most popular languages for them happen to be sh/bash/zsh/fish etc, which make a different set of tradeoffs than the 'traditional' programming languages.
I'm only partially aware of the history standpoint. My understanding is that before awk, shells (sh, mostly, IIRC) were about the only scripting level languages on various Unixen.
The language has evolved with compatibility since the original release in 1971, before it had loops, conditionals, or functions. That's before yacc and probably before some common parsing algorithms were even invented. It is a proper programming language by now, but it just feels old and crufty.
I think it needs a few more OS bindings as well. The "test -f / -d" stuff is a little impoverished.
Previously discussed two years ago here: (https://news.ycombinator.com/item?id=9207738)
[edit]
Found it here:
I'm aware of, like, TermKit[0], but that didn't get finished and I don't recall hearing about any newer efforts.
You can also see a list of other shells on the GitHub wiki page titled ExternalResources: https://github.com/oilshell/oil/wiki/ExternalResources
I'm focusing on the shell as a programming language at first, in contrast to TermKit.
Is that developer are THE WORST. Them will refuse to switch to anything better if it means to a small change in his old ways.
That is why C/C++/Js/Bash rule the world...
However, still developers will not buy it. I think maybe it could work for a subset of the user that need to work with data-science and subset of system automation.
Last I checked I think the thing I'm missing from using iPython purely as a shell is the ability to pipe?
Adoption would be even harder, but any resulting language is not going to look like Python or any other "mainstream" language, if it's intended to actually be usable.
Oil is parsed line by line like any other shell, and there will be a simple rule to determine whether you're in command mode or expression mode.
The "lexer modes" technique (previous referred to as "lexical state") should handle the two sublanguages easily. You basically need it to parse bash, and you will need much less of it to parse oil. My bash parser has 13 lexical states (up from 8), but I expect oil to have somewhere between 4 and 8.
How it's used in the bash parser: http://www.oilshell.org/blog/2016/10/19.html
https://news.ycombinator.com/item?id=13478291
It is a design issue to combine command and expression syntax (unquoted and vs. quoted literals). But I think I've figured it out ... we'll see in a few months! The blog will have updates on these kinds of issues.
Once you upgrade it's of course not POSIX compatible anymore :)
The takeaway after all that was, rarely is one language better than another, its really about trade-offs.
And frankly the older ones are a lot better at following the KISS principal and actually providing an uncluttered framework for writing _EFFICIENT_ code. For example, everyone knows that Java is a "better" and more productive language than COBOL, yet when you start talking to people trying to port old COBOL apps to java, what you discover is that it turns out that COBOL might actually do a better job in the business/transaction space. Same with python and Fortran in the HPC space, and strangely enough the core infrastructure all these more modern efficient languages are running on tends to be "C" based. Which despite the laundry list of issues that people like to parade around, seems to solve systems programming problems better than anything else.
None of this would really be a problem except for the fact that the new up and coming language that is cool this year changes from year to year leaving a wasteland of abandoned poorly maintained projects using languages that no one really knew, and are despised by the people who are hired and have to learn them for that one job before moving on to the next one.
So, yes bash is a POS, but its universal at this point, isn't really that bad at solving the core problems its designed to solve, and any developer with 1/2 a brain can write a sys V init script (or a lot of similar things) with little more than a couple hours and without the baggage of yet another 1/2GB language/framework/etc pulled in to sit alongside the dozen others already in the project.
</rant> Now get off my yard.
Which is better than the days when csh and different Bourne flavors were common.
Fish does have bugs though. I've seen enough stack traces while running early builds to question the security (although I'm sure we all remember the bash shell-shcok bug that was laying around for years).
If you haven't tried fish, I highly recommend it. It takes a little getting use to, but overall it adds a lot of really amazing functionality that I find really useful.
Serious question (that I've asked elsethread): what does fish offer than zsh & bash lack? A quick perusal of the docs reveals nothing obvious, but … I could be missing stuff.
* Instant auto suggestions, you just type stuff, like in a web browser.
* Sane defaults, no need to bring around your own config. On a new machine I just install fish, run fish_config and customize the prompt with the web UI.
* Scripting with simple syntax, especially for one-liners. Yes, fish is incompatible with bash, but it's less of issue in this case.
Some features are plain mistakes, but some of them are useful things that newer shells aren't taking into account (particularly with regard to file descriptors). I'm aiming for a superset of bash semantics, but with different syntax. Importantly, it will be backward compatible, because bash can be converted to oil automatically. (I'm working on that now, not released yet.)
http://www.oilshell.org/blog/2016/11/09.html
The rest of the blog has a bunch of shell corner cases, like:
http://www.oilshell.org/blog/2016/11/06.html
http://www.oilshell.org/blog/2016/10/28.html
Although I probably only blogged about 20% of the weird stuff I found.
I'll add that my ideal "command and control language" (as I've been thinking of it) would also be one that's strong at expressing data literals. If there are a number of data types like maps, sets, lists, etc., then it should be easy to initialize them with literals (not true in languages like Java). There are so many configuration file formats that I wonder if they could be subsumed into files with literals expressed in this language (perhaps with a "data mode" to prohibit executable code).
I've been wanting to design a language along these lines as well. I'll start by learning about oil, and reach out if I'm interested and able to contribute!
There is also going to be influence from R and CSV files (which goes with awk).
I have thought about the config problem a lot -- and a configurable sandboxed "data mode" you are talking about is probably what I will go with.
If you squint, the oil shell will look not unlike nginx configs, e.g. a bunch of words with nested {} for blocks. Maybe like https://github.com/vstakhov/libucl or https://github.com/hashicorp/hcl (hm this seems to even have here docs!)
This kind of thing was done a lot with Python at my last job, to various degrees of success (e.g. https://bazel.build/ - the build language is derived from Python). Python sort of has some "data mode" sandboxing features, like being able to set __builtins__ when you exec/eval, but probably not enough. It didn't work well enough to prevent people from writing their own config file parser eventually. The syntax is close but not exactly what you want for some use cases.
The short answer is that the experiment didn't work very well. If you want a shell based around Lisp, the new "oh shell" is closer to that. It has homoiconic syntax, which I don't agree with. I talked about that a bit here:
http://www.oilshell.org/blog/2016/10/28.html
The oh shell is influenced by the es shell, which was almost literally a Lisp. There is also EShell (ELisp) on my Github wiki, and earlier Scheme shell. So without going into too much detail, I think the idea has been tried and it has failed not by accident, but for fundamental reasons.
That's not to say that Lisp isn't hugely influential. Julia and Elixir both have very Lisp-like metaprogramming which I hope to take some inspiration from. Without having tried it in detail yet, I think the lesson is that in 2017 you don't need homoiconity to have metaprogramming. And another lesson is that people like syntax (me included!)
The shell especially needs syntax because it will often be typed in a terminal as opposed to in an editor with help.
Specifically, maps as association lists are very natural in lisp. So are many other structures. And JSON is easily seen as a crappy sexp. XML is easily a complicated sexp. Python's pickling is a complicated eval.
And I get it. Some things have a more "natural" syntax that isn't postfix. Even in common lisp, the reader macro is often used for infix things. However, when the examples used are easily translated to sexps, and back, it makes me wonder why we don't think programmers could learn to write them.
I mentioned in that blog post that Clojure introduced meanings for [] and {} in a Lisp, and I generally view that as a good thing.
Also if that syntax is only used for data literals, and not code, then it loses the whole Lisp paradigm and I wonder why you would want that syntax.
Lisp was using [] and {} before in a lot of software.
For example this is a data structure in Connection Machine Lisp called Xapping, basically a parallel hash table:
{moe->"Oh, a wise guy, eh?" larry->"Hey, what's the idea?" curly->"Nyuk, nyuk, nyuk!"}
Scheme was using [] also many years before in the standard, programs and books.
In this example, though, what was the advantage of the special syntax? (I'm assuming there was one. Just a lot of these parallel hash tables written in literal form?)
Which is wrong. It's just that Lisp syntax looks and works a bit different. Especially there is a data syntax for s-expressions and the syntax for the programming language Lisp is defined on top of s-expressions.
> And that they should avoid the reader macro at all costs.
Common Lisp itself uses read macros for its implementation. Applications use them in various ways.
> In this example, though, what was the advantage of the special syntax?
Xappings were a central data structure in Connection Machine Lisp, thus it's not unusual that it had a printed representation.
For the representation.. So this was not simply a literal syntax, but a serialization one. Which makes sense. And I can see why the serialization syntax is easily usable as a literal one.
Lisps don't have anything that even comes close.
My question is why does the teaching often go that there is practically no syntax for lisp. Now, it is often followed quickly with "in lisp, you can write your own language."
But then, those two thoughts are hard to justify next to each other. Again, just see this thread. The complaint is that there is not enough syntax in lisp. The argument appearing that you have to have another language to express some ideas. And again, I'm not even disagreeing with this claim. Just trying to understand why it has the foothold that it does. That you drop to lisp if you want to use no syntax.
See for example the syntax which is implemented by macros like LOOP or ITERATE. The LOOP syntax is documented in the ANSI CL spec...
http://www.lispworks.com/documentation/lw51/CLHS/Body/m_loop...
Usually, the syntax is already parsed when it comes into a macro.
E,g:
(sentence-macro subject object verb)
Though this is still syntax, it is parsed syntax.
When we say that Lisp has no syntax, it means that it dosen't have that silly, counterproductive stuff naively imitated from natural languages which represents the tree in a way that requires mind-bending work to recover.
(loop for i from 0 to 10
for j from 10 downto 0
collect (cons i j))
LOOP is also not universally loved in the Lisp world. It's fine if you don't need to extend it. Not only is it inherently inextensible (for no good reason), but it's hostile to wrapping. If you write a MYLOOP macro which passes most of its arguments to LOOP and adds a few clauses of its own (translated to LOOP syntax), you have to parse the LOOP syntax to see where your clauses are and in which order in relation to the standard ones. (Unless you do something hacky or ugly, like recognize your extensions by some delimiters, without regard for the surrounding syntax, so that they don't actually blend in.)
LOOP is the only high level control construct I've ever seen anywhere in which you can achieve nonportable behaviors: because of the construct itself, not because of nonportable expressions plugged into it.
The treatment of clause symbols as character strings, so that CL:FOR and MYPACKAGE:FOR can both head off a for clause is a design smell. LOOP should have been designed to use symbols in the CL package, so that if someone wants to use APPENDING, they nicely import that symbol or pick it up via :use.
LOOP asks you to use different syntax for walking a vector and list, when the rest of Lisp has a sequences library that lets you use common functions for both. Supporting the specialized syntax is fine (just like the library also has list and vector specific things), but the lack of a generic sequence iterating clause shows a discord with the rest of the language. It didn't require much imagination to have, say a FOR x OVER <list-string-or-vec>.
We can have a perfectly good Lisp dialect without any sort of mini-language parsing macro like LOOP. It illustrates what you can do with macros, not what is usually done with macros. Macros usually leverage their tree structure and destructuring not to do any parsing work, and focus on the transformation and its semantics.
Various LOOP implementations are extensible.
> Macros usually leverage their tree structure and destructuring not to do any parsing work, and focus on the transformation and its semantics.
Untrue. Many macros need custom destructuring or even walking the code in some way.
> We can have a perfectly good Lisp dialect without any sort of mini-language parsing macro like LOOP.
Sure: basic Scheme, but then Scheme has its own complex looping constructs like http://wiki.call-cc.org/eggref/4/foof-loop .
Iteration macros like LOOP, ITERATE, FOR and others are very convenient and powerful.
Destructuring is just how we access the tree (already parsed object). Sometimes the tree isn too complicated for the simple destructuirng performed by destructuring macro lambda lists, so we have to do things like walk substructures and apply DESTRUCTURING-BIND or use some pattern matching library or whatever. Example: just because I I have some MAPCAR over a list of variable-init pairs (qualifies as "custom destructuring") doesn't mean I'm parsing syntax that hasn't been parsed. I'm just destructuring a syntax tree that hasn't been destructured. Destructuring isn't parsing. Destructuring is not even mentioned in papers and textbooks on compilers; it falls into the bucket of somehow walking the tree, which falls under semantic analysis.
Code walking is ultimately done to the expansion of every macro. COMPILE and EVAL perform code walking. No textbook on compilers will refer to code walks over an AST as part of the parsing stage.
Of course the point is valid that Lisp macros sometimes take a flat list of items and apply recursive phrase structure rules to recover a tree (or small subtree, as the case may be). This is rare, and basically a last resort device; if you're doing that, you're writinig some sort of "big deal" macro. Instances of it are rare in Common Lisp, and I don't suspect it is done frequently in Lisp programs. It's great that wan do that. It's also great that because of the way the langauge works, we can accomplish a lot without having to do that.
Speaking of ITERATE, its parsing of clauses is trivial, because it uses Lispy syntax. Dealing with (for var = expr) versus (for var initially expr then expr) versus (for var first expr then expr) is "quasi parsing". OK, we have a FOR. What isn the third item? Switch on a bunch of cases: =, INITIALLY, FIRST, .... If it's unrecognized, then signal an error. In each of these cases, certain things are fixed by position in the syntax already. This is "parsing" only in the sense that Unix people refer to simple trivial line argument processing as "parsing". (Only a few utils do actual parsing or a recursive grammar, examples being find and tcpdump.)
That's not syntax. Syntax is concerned whether a sequence of words are valid expressions in a language and determines syntactic categories for these. You can lookup a better definition of syntax, I'm too lazy.
These are all DEFUN forms. Some are valid Lisp, some are not.
(defun foo () ()) is valid
(defun () foo ()) is invalid
(defun () () foo) is invalid
(defun (foo) () foo) is invalid
(defun foo () foo) is valid
(defun () foo foo) is invalid
(defun foo foo ()) is invalid
(defun (setf foo) (setf foo) foo) is valid
(defun foo (setf foo) (setf foo)) is invalid
(defun (setf foo) foo (setf foo)) is invalid
(defmacro defun (name args &body forms) ...)
> sentence-macro doesn't have guess what part of the utterance is the object and which is the verb. The verb is the third argument, and that is that.
Macros implement more complex syntax. Check the ANSI CL specification and its EBNF syntax declarations some time.
> When we say that Lisp has no syntax
Then it's just wrong and misleading.
Take the DEFUN macro:
the EBNF (extended backus naur form) syntax definition for DEFUN is:
defun function-name lambda-list [[declaration* | documentation]] form*
(defun (bla blub) (&rest foo &optional bar)
(declare fixnum) "oops" ((((fooo))))))
The macro has to check that. It better rejects invalid expressions. It also has to look at the elements of the form to destructure them in the right way, so that it can process them and create a new valid form.
The Lisp implementation provides for the implementation of DEFUN as much as:
(defmacro defun (spec args &body body ...) ...)
Questions the macro has to answer:
* is the function name a symbol or a list of the form (setf foo) * is the arglist a valid lambda-list? Now check the EBNF syntax for lambda lists with whole/optional/key/rest options with default values and what have you. * now it has to parse the body: * is the declaration valid? Now check the EBNF syntax for declarations to see what needs to be done. * is the documentation a string at the right position? * is the body a sequence of forms? Now check the EBNF syntax for FORM.
This all has to be backed into the DEFUN macro somehow or checked from there. And not all implementations are good at it.
Various syntax errors:
The name is not valid:
CL-USER 14 > (defun (foo bar) baz (list))
Error: (FOO BAR) is neither of type SYMBOL nor a list of the form (SETF SYMBOL).
1 (abort) Return to level 0.
2 Return to top loop level 0.
CL-USER 16 > (defun (setf bar) baz (list))
(SETF BAR)
CL-USER 17 > (compile '(setf bar))
Error: Invalid lambda list: BAZ
CL-USER 19 > (defun foo (&key ((aa))) (list))
FOO
CL-USER 20 > (compile 'foo)
Error: Malformed (keyword variable) form in &key argument ((AA))
CL-USER 24 > (defun foo (&key (aa)) (declare inline-function foo))
Error: Alist element INLINE-FUNCTION is not a cons or NIL
CL-USER 26 > (defun foo (&key (aa))
(declare (inline foo))
(((lambda ()
(lamda () ())))))
FOO
CL-USER 27 > (compile 'foo)
Error: Illegal car ((LAMBDA NIL (LAMDA NIL NIL)))
in compound form (((LAMBDA NIL #))).
An INFIX macro:
(infix 3 + 2 ^ 5)
If it is still not clear, below is the syntax for the LOOP macro. The LOOP implementation has to implement the syntax, so that
(loop for i below 70 do (print i))
(loop do (print i) for i below 70 )
The ``simple'' loop form:
loop compound-form* => result*
The ``extended'' loop form:
loop [name-clause] {variable-clause}* {main-clause}* => result*
name-clause::= named name
variable-clause::= with-clause | initial-final | for-as-clause
with-clause::= with var1 [type-spec] [= form1] {and var2 [type-spec] [= form2]}*
main-clause::= unconditional | accumulation | conditional | termination-test | initial-final
initial-final::= initially compound-form+ | finally compound-form+
unconditional::= {do | doing} compound-form+ | return {form | it}
accumulation::= list-accumulation | numeric-accumulation
list-accumulation::= {collect | collecting | append | appending | nconc | nconcing} {form | it}
[into simple-var]
numeric-accumulation::= {count | counting | sum | summing | }
maximize | maximizing | minimize | minimizing {form | it}
[into simple-var] [type-spec]
conditional::= {if | when | unless} form selectable-clause {and selectable-clause}*
[else selectable-clause {and selectable-clause}*]
[end]
selectable-clause::= unconditional | accumulation | conditional
termination-test::= while form | until form | repeat form | always form | never form | thereis form
for-as-clause::= {for | as} for-as-subclause {and for-as-subclause}*
for-as-subclause::= for-as-arithmetic | for-as-in-list | for-as-on-list | for-as-equals-then |
for-as-across | for-as-hash | for-as-package
for-as-arithmetic::= var [type-spec] for-as-arithmetic-subclause
for-as-arithmetic-subclause::= arithmetic-up | arithmetic-downto | arithmetic-downfrom
arithmetic-up::= [[{from | upfrom} form1 | {to | upto | below} form2 | by form3]]+
arithmetic-downto::= [[{{from form1}}1 | {{{downto | above} form2}}1 | by form3]]
arithmetic-downfrom::= [[{{downfrom form1}}1 | {to | downto | above} form2 | by form3]]
for-as-in-list::= var [type-spec] in form1 [by step-fun]
for-as-on-list::= var [type-spec] on form1 [by step-fun]
for-as-equals-then::= var [type-spec] = form1 [then form2]
for-as-across::= var [type-spec] across vector
for-as-hash::= var [type-spec] being {each | the}
{{hash-key | hash-keys} {in | of} hash-table
[using (hash-value other-var)] |
{hash-value | hash-values} {in | of} hash-table
[using (hash-key other-var)]}
for-as-package::= var [type-spec] being {each | the}
{symbol | symbols |
present-symbol | present-symbols |
external-symbol | external-symbols}
[{in | of} package]
type-spec::= simple-type-spec | destructured-type-spec
simple-type-spec::= fixnum | float | t | nil
destructured-type-spec::= of-type d-type-spec
d-type-spec::= type-specifier | (d-type-spec . d-type-spec)
var::= d-var-spec
var1::= d-var-spec
var2::= d-var-spec
other-var::= d-var-spec
d-var-spec::= simple-var | nil | (d-var-spec . d-var-spec)
Arguments and Values:
compound-form---a compound form.
name---a symbol.
simple-var---a symbol (a variable name).
form, form1, form2, form3---a form.
step-fun---a form that evaluates to a function of one argument.
vector---a form that evaluates to a vector.
hash-table---a form that evaluates to a hash table.
package---a form that evaluates to a package designator.
type-specifier---a type specifier. This might be either an atomic type specifier or a compound type specifier, which introduces some additional complications to proper parsing in the face of destructuring; for further information, see Section 6.1.1.7 (Destructuring).
result---an object.If you look at these files it's mostly declarations and one-line functions. It's not even doing anything. The lexers and parsers are a whole other story, certainly greater than 19K lines combined.
~/src/cfe-3.8.0.src/include/clang/AST$ wc -l *.h
2742 DeclObjC.h
2809 RecursiveASTVisitor.h
2927 DeclTemplate.h
3198 OpenMPClause.h
3249 DeclCXX.h
3800 Decl.h
4154 ExprCXX.h
4942 Expr.h
5723 Type.h
61525 totalMy gripe with the second will be simply that you will retraverse a ton of corner cases and other encoding issues in order to have your literal syntax. More, I will probably be required to have yet another batch of tooling to support programmatically generating these literals.
At the end of the day, probably won't matter. Odds of success here are already low. So, I can see siding with emotional aesthetics for a syntax. I just face palm with all of the tooling that goes out the door simply because people are averse to parens.
I don't really see a paucity of tools for JSON or Python, unless there is something special you're looking for.
The larger issue of metaprogramming is important, but that would require code to be represented uniformly too, not just data. I'm looking into Julia and Elixir metaprogramming, which take advantage of a uniform Lisp-like representation, but also have rich syntax.
I would say it's 2017 and we should be able to have BOTH syntax and metaprogramming/tools. I have a lot of posts about about parsing on my site. If there's a problem with parsing something, I prefer to fix the parsing tools than to mangle the language's syntax to fit an ancient model.
The year is just an appeal to emotion. It would be nice if we could have and eat cake. Empirically, things haven't shaken out in that direction. And it wasn't long ago that there were a host of face palms for JSON parsing out there. Things are getting better, in some regards, but adding another JSON like parser to the fold will only make that worse.
Yes, you can inline lisp expressions. But, I constantly paste in standard shell exports and they work exactly as you would expect them to.
Again, we agree that lisp is not necessarily a good shell language. Just, EShell seems a poor example to look at when making this argument.
Edit: Where is the wiki you referred to? In particular, where you discuss eshell.
And I realize I made an omission in my post. I meant to say not just learnings, but use. I would think the biggest hurdle for you will be building a user base.
I'm want to publish the test matrix and get some help because filling out the corners takes a lot of time. The general architecture is pretty sound though -- I hit all the "major areas" of shell, although I may need to make another pass at globbing.
If you are adventurous you should be able to clone it, follow the instructions for ./pybuild.sh. Then ./spec.sh install-shells, and then run './spec.sh smoke'. That test should pass, but there are like 15-20 other categories in spec.sh in various states of disarray!
Of course this is in Python, but to me it makes sense to fill out a sketch before porting it to C++ (only the runtime; I'm hoping to bootstrap the parsers in oil itself). Not sure if that will inhibit or help contribution.
command arg1 arg2 ...
What feels ancient is the lack of good parse error messages and runtime error messages in bash.
And syntax like a=X but not a = X or a = 'X' -- the difference being the spaces.
And (echo hi) is OK for a subshell, but you need { echo hi; } for a group -- space and ; are required in the latter case, but not the former.
The lack of proper functions in shell also feels ancient. For example, writing a completion function in bash is a pretty horrible experience. As another example, a simple function to escape HTML is either 'sed' or three statements like ${foo/&/&}, both of which are obtuse.
I have a minimal Unix setup, with lots of little programs that do one thing and do it well. For example, mutt is my MUA (mail user agent). It just reads my emails, but fetching or sending is carried out by other programs. Fine.
Now, the issue is that every little Unix utility that is non-trivial comes with its own little DSL for configuration. Most are inflexible, naive and different to each other. It's hard to script mutt. If you want to do some stuff the author hasn't planned for, you are better off patching the code. Flexibility, and composability, is not as good as they preach.
For this reason, I am migrating everything to emacs. It offers me what Unix only promises but falls short at. Everything is an elisp function and thus easy to combine. Things are more organic. And still minimal.
" We really are using a 1970s era operating system well past its sell-by date. We get a lot done, and we have fun, but let's face it, the fundamental design of Unix is older than many of the readers of Slashdot, while lots of different, great ideas about computing and networks have been developed in the last 30 years. Using Unix is the computing equivalent of listening only to music by David Cassidy."
Rob Pike, https://interviews.slashdot.org/story/04/10/18/1153211/rob-p...
But overall, I am more comfortable on live environments like Smalltalk or Emacs. There's just one fundamental abstraction, and things can be composed beautifully.
In Unix, on the other hand, everything is not a file. And even if it was, little applications like mutt are silos. The code is static, it can't be easily changed or opened. A file interface, in the form of STDIN/STDOUT is too unstructured.
After Amiga, Atari, Acorn and Windows, getting to learn UNIX ideas seemed quite interesting (I started with Xenix).
However, once at the university I got the opportunity to dive into the work of Xerox PARC and ETHZ, the more I learned about them, the more I came to realize that I actually cared more about their vision about how computing should look like, than anything I ever used in UNIX variants.
Of course, Plan 9, Inferno and C+@ are other Bell Labs ideas that would have improved the overall UNIX experience, but as you say, have been ignored.
let (foo bar)
ls foo "bim/baz" ${foo}
In fact, with a minimum of work I think this could be turned into a Lisp shell …
I'm not sure how TCL and the Thompson shell relate though.
And my list is not exhaustive, either. But it makes the point to those that would erroneously dismiss this (as you yourself do) as something that "doesn't see much use today", that this, given how long the list of non-daemontools-family commands that operate this way actually turns out to be when one looks around, is not really a daemontools, or even a "Bernstein", thing.