Crush: A command line shell that is also a powerful modern programming language
github.com
github.com
Then your environment is useless to me. Because I have better programming languages which, despite being able to interact with the system and its commands, don't pretend to be one with the system. As a result, the core high-level language constructs mesh better, because they're not trying to serve two goals at once. It's not possible for something that's not a 'real' shell (including fish, zsh, even csh) to integrate with the system the way a real shell does, because the system itself was designed around the shell.
# ls | grep foo
This involves two programs written in nicer language (C).
# ls | rg foo
This involves two programs written in nicer language (C & Rust).
Feel free to come up with JS program that does something well and/or is usable in pipeline. I think you're missing the point of the shell.
Perl code is often written once and then runs for years without issue. I have on occasion fixed broken Perl scripts despite having practically no experience with the language. It wasn't a problem at all, because your average maintenance job is "something was moved" or "we're switching out X for Y", but not "we need to rewrite this complex system that is a Perl script for some reason".
I would intuitively agree with you as there are hundreds of unnoticeable perl scripts doing their job, but as someone with moderate experience with the language, I can say that when I did maintenance tasks, they were mostly the second kind.
Tcl is horribly, terribly under-appreciated. It is one of the great might-have-beens of modern computing, almost as much as is Lisp.
And I think Lua is more elegant than TCL overall. TCL's stringiness really gets clumsy in certain types of programs, and it's a much larger target than Lua overall.
Not to say that TCL is bad, but I'm glad that dict-based languages prevailed over TCL's string-based-ness
I especially see this being useful in my CI environment which has a handful of functions, some json and yaml parsing (using jq and python respectively), etc. It's a little cumbersome at the minute but not so much to switch from bash and the rest of the project is JS anyway so it would be good to be consistent.
> useful in my CI environment
For sure. I've been meaning to play around with getting a docker image with the shell set as the default. It's definitely a use-case I had in mind.
>> The point of shell is that it interoperates freely with all programs and all programs share a language.
Also doesn't mean people should stop trying to come up with these new shells. May be some "new" shell will catch up and tomorrow's "all" programs will be compatible with that shell. For example, now on Mac OS X, now ZSH is the default shell.
What do you mean by can't? If it misses some mode of interaction, that can be always added.
Sometimes the inconsistent solution will be simpler by moving the complexity to the choice of the mode of interaction. Sometimes not.
"ls" lists files. Sure. Simple interactive command. But, thinking about this, "echo " also lists files. "ls" provides information about files. Sure. Simple interactive command. But, "stat" specifically provides information.
So, I would reach for
stat --format "%n %a"
To get name and octal permissions. You are correct, and wrong at the same time. Easy enough to accomplish -- but using * instead of ls. Note that the "space in filename" becomes the big issue (use %N in stat format).
So your conclusion is 100% correct. Thank you for your post and insight
ls -l | awk '{print $3}'
ls -l | cut -f 3 -d ' 'I wouldn't hold my breath for something else to replace sh/bash as the default shell on the major nixes within this century. Pessimist? More like realist :-)
There are more young people who know Perl or Tcl or Lisp than know Crush. I've never understood this reasoning - "Hardly anyone knows X. Let's replace it with something nobody knows." At most, you're making an argument about a lack of sunk cost.
> May be some "new" shell will catch up and tomorrow's "all" programs will be compatible with that shell
Think of it like transitioning to life in a wheelchair. You can't "catch up" with your house - it just wasn't designed for wheelchair use. Renovating all the narrow corridors, random steps, tight angles and vertical reaches probably costs more than rebuilding.
In this comparison, the mobility mode is your shell.
Because there's a value proposition. Just like TCL, Perl, and Lisp had value propositions over other available languages. It's fine if you don't see the value proposition in Crush but progress sometimes forces us to learn new things instead of sticking with what we have.
If I’m being honest, you’re not adding anything constructive at all here - it seems you’re in a competition with yourself as to how much you can degrade Crush for whatever reason.
You act like it’s a choice between Crush and Perl. People don’t have to learn Crush. People don’t have to learn Perl. Will picking between the two have any substantial effect on one’s career outcomes these days? Probably not, as candidates will have already learned the important languages for getting a job anyway.
But it is cute to couch it in a complaint about not being constructive; points for shamelessness.
By that logic no one should make anything new to replace the incumbent.
In case of Perl, PHP came and it replaced Perl as mainstream web development language.
In sysadmin stuff where Perl used to be mainstream, now significant chunk is taken away by Python.
These days JavaScript appeals to younger people more. Times change and with it the tools people use also change.
There is no guarantee that any new project will succeed but you won't know unless you try.
You're really bringing up a separate point which is "should we be reinventing the wheel?", I think that's a fair point, but it's not a response to the original commentator.
I don't know perl and I'm not young, but I do understand and appreciate the role perl -e plays as an extension of bash. When I run into an issue where I need more than bash, but less than python, I'll search around SO for a perl -e command for what I need.
This is often how people decide to learn a tool: they have a problem for which it is the best solution. They get some exposure to the capability and slowly expand on their knowledge.
The "ecosystem" of programming languages is healthy. There are many options, some to master, some to consider. But not knowing how to use a tool -- especially a widely-used and powerful tool -- is no badge of honour or discernment.
but what if people don't know how to learn?
instead of getting wrapped up in all of this 'knowledge preservation' nonsense, shouldn't we just mimic what they are already doing and assume it to be better, because the activity occurs in a year number that is higher than the previous behavior pattern?
now that would be real progress.
or, not.
Societal results sadly noted.
This presupposes that the Unix shell is the way it is because of careful, considered, forward-thinking design, rather than being a haphazard heap of mouldering hacks upon lazy corner-cutting upon quick-n-dirty bodges that should’ve been thrown out and replaced decades ago.
I agree that trying to improve the *nix shell is a bad idea… because the _Unix shell itself_ is a bad idea. Fifty years is long enough. Instead of bending over backwards for that obsolescent garbage heap (for which no-one will thank you, as we see), study it, learn its lessons, and draw a line under it.
Then go build a clean, modern shell architecture from scratch, using modern tooling and techniques, unhobbled by the myriad defects and deficiencies of the old. It’ll take a tenth of the time and be ten times better (if not, you’re still doing it wrong).
See also:
https://www.google.com/search?q=sunk+cost+fallacy
https://homes.cs.washington.edu/~weise/uhh-download.html
--
p.s. As for driving early adoption, don’t even think of trying to win over the Unix old guard (who will only bring regressive practices and complaints with them), but sell it hard to the new generations of users who are accustomed to modern UI/UX standards and much less impressed by what the “Old Unix” dogpile has to offer.
Windows wasn’t designed around the command prompt, it wasn’t designed around shell utilities. When I call “cat”, it’s not important that I’m writing those three letters, they are a library way of calling a set of file open/read OS functions. When I write “ping”, it’s not the ping tool I care about, it’s the network api calls it does behind the scenes. The Unix/Linux shell is then seen like a “Python standard library” of ways the user can control the OS, and it’s a hugely inconsistent, undesigned, ad-hoc agglomeration of names and parameters and bodges, which cheats out on dealing with the complexity and instead offloads that straight on to the user in a user hostile way.
What powershell tries to do for Windows is make a somewhat designed, consistent, and more user-approachable set of wrappers around the Windows OS system calls, along with guidelines and support for vendors to build in that style, a human interface that is introspectable, built to be programmable, but still composable and modular.
The objection that trying to do such a user-focused thing on Unix makes a “useless” system is weird because that - a user interface to tools/library of OS call patterns - is what the shell is. In what way is “resolve-dnsname” pretending to be one with the system that dig and nslookup aren’t? They’re names which trigger a pattern of network traffic and show a result. In what way is piping text from dig to sed “real” and piping resolve-dnsname to format-list “pretend”?
In what way is running “docker compose” from sh real, but running it from zsh is pretend?
In what way is it a better design to have (a pile of shell tools and Python library wrappers and Python wrapping shell tool calls), instead of (a Python based shell with transparent shell tool calls)?
Like, “I want what I’m familiar with, too many things to relearn”, or “too many things are awkward in this approach (cough PowerShell)” are reasonable objections, but “trying for a more consistent, more human friendly, interface with more high level language constructs at one’s fingertips makes a thing useless to me” feels heel dragging or Stockholm syndromeish. Presumably you don’t object to there being a choice of multiple command line tools to do a task? Why object on principle to a choice of other OS “front end experiences”? Is there any version of things which you can imagine as being both “better” (less arbitrary and inconsistent than shell) and “not pretend”?
> “The point of shell is that it interoperates freely with all programs and all programs share a language.”
What language? Some treat piped bytes as bytes (e.g. piping a binary file), others as ASCII, others as UTF-8. All support different parameter options, names and values and ways of specifying them. Many handle output formatting in different ways, many trigger internal states from environment variables. If the point is a common shared language, the point doesn’t seem to have been achieved from a user perspective. And if the point is that “I have to suffer a system that’s tied to a shared language with an AIX box in a bank basement last touched by a human in 1997” the point is a bad one. I use my computer a lot more than I use AIX and I have more computing power to spare on human comforts and conveniences.
The UNIX philosophy was to write a solid system by taking most of the hard problems in kernel space and kicking them into user space. This is not a slur, this is a description. Especially at the time, it wasn't at terrible solution. I kind of think of it the same way as many programming languages have observed that "the standard library is where things go to die"; the code in a system that is hardest to change, well, it doesn't change! So any bad decision that ever gets in there is almost permanent. Part of the reason UNIX won is that it enshrined fewer bad decisions in the kernel than its competition did. I think you could even argue it had more bad decisions overall than its competition, at least towards the beginning; it's just that far fewer of them were enshrined in the kernel.
This is also the philosophy of the pipe. It disclaims all layout and moves that responsibility on to the consumers. There is no "shell language", at all. This is, simultaneously, a big strength and a big weakness.
One aspect of that is precisely that the strength of this approach tends to inhibit anyone who tries to beat it with something like Powershell. All Powershell had to be was better than any shell on Windows. This was not hard. For something like Crush, to take off it has to be better than any existing shell on UNIX, and as annoying as unstructured pipes may be, the ecosystem as a whole still has a lot of positives about it, because it's been tuned for 50+ years. It's going to take a lot of work to match that, and so far, it doesn't seem like any of these projects are achieving take-off velocity.
Arguably, the best thing to do would be for all these projects to get together and try to define a standard UNIX-level non-textual rich serialization of some sort, using lessons from Powershell and arguably things like D-Bus to do it, because none of these projects individually seem to be on a success trajectory.
Who cares about "taking off"? If it's useful for 10s of users, it has taken off.
Consider xonsh (https://xon.sh/). I don't think it will ever be "mainstream" the way bash or zsh are. I doubt it will ever come preinstalled in any major Linux distribution. But it's used by enough and has enough momentum that I don't worry it will die. That alone is enough for me to use it as my primary shell. I don't care that xonsh will never "take off".
It doesn't have to be the goal of the project or the author, but there are real advantages to achieving a takeoff, even to the users of a product.
What is the one thing that GNU ls does well, by the way?
Backwards compatibility with itself, probably.
If you came away from my post with the impression that this system produces a lot of elegant software, well, I must have screwed up in my writing somehow. What it produces is bodges upon bodges, which are then polished by years of ramming into the corner cases and putting in another bodge to fix them. There is actually something to be said for this design methodology, and it's not all bad, but it rarely produces elegance.
Now, at least GNU ls doesn't send or receive email or something. A quick scan of its man page doesn't show many any features grossly out of line with its primary task. But yeah, it has a lot of features for sure, many of which nominally overlap with tasks that could be done by other tools (sort, ignoring backups, -D ("generate output designed for Emacs' dired mode", lol), quoting styles...).
I feel much the same way about Python. Very little of its philosophy expressed back around 2000 or so is in effect today. And there's no one release you can point at and say, "this is it, this is when it stopped being simple", yet, clearly, today it is no longer simple.
The "UNIX philosophy" gets cargo cult by younger generations thanks to a couple of statements in a few worshipped books, yet there is hardly any UNIX clone post V6 where it was actually followed upon.
I know that Crush currently doesn't interact very well with external commands, but I don't see where that is an inherent restriction in the design rather than a missing feature.
I think the fact that you can already use this with most programs and it is never less powerful than bash seems like good indication that the system doesn't need to be reworked around this shell. It fits in cleanly to existing paradigms.
A lot of people seem to have negative opinions about Tcl (maybe because it's more of a LISP than its syntax seems to imply), but as someone who is just discovering it, I'm shocked at how powerful it is for how unpopular it seems to have become. It's like some kind of unexpected cross between Bash, LISP, and (with Tk) Visual Basic. There's something about it that's just kind of fun.
I think some of the pushback might be because people tried to use it to build large applications and failed. Languages like Java and (dare I say it) Visual Basic were more comprehensive and had good IDE support.
Yet Tcl tends to be forced on undergraduate programmers, who probably just learn to loathe it. I lost several toes to it myself decades ago, and haven’t really felt inclined to pick it up again.
That's not the idea of shell. The idea is that I write programs in Python, JavaScript, R, and C++ regularly, and about 10 different DSLs (SQL, HTML, etc.) And I work on systems written by others, consisting of even more languages.
I need a language to glue them together. A language to express build automation and describe deployed systems. Most big systems consist of more than one language.
Shell is the best language for that, but it's also old and crufty, with poor implementations.
When you program in shell, gcc, git, pip, npm, markdown, rsync, diff, perf, strace, etc. are part of your "standard library".
* Preserve compatibility with popular shells like bash.
* Address some of the broken parts in a way that makes the world better. Quote handling is the original sin of *nix shells. If somebody fixed that you would have my full attention.
Otherwise I can just stick with Bash + Python3.
In order to preserve compatibility, you must handle quotes, namespaces, etc in the same way as Bourne-derived shells, because they will be expected to run scripts that use nested interpolation, or string declarations without quotes, or whatever. Shells that use a more robust syntax, like Fish or Powershell, can't ever be compatible.
If Apple shipped some Swift-powered shell and all of their management documentation started using it, I'd bet a non-trivial portion of administrators would start to use AppleSwiftShell (although I do deeply appreciate that's a severely flawed comparison since macOS already has a sane functioning shell, but it's the best I could think of as a huge company that owns the platform pushing such a concept down)
Its doing good because it is good and people started to notice once MS propriatery chains went down.
Scripting languages like Python, Ruby, Perl, etc., have dealt with upgrade many times. It has not always been pretty, but the alternative is stagnation.
https://news.ycombinator.com/item?id=24086213
And yes I drafted a blog post called "Unix Shell Should Evolve Like Perl 5" which says exactly what you are saying:
http://www.oilshell.org/blog/2020/07/blog-roadmap.html#the-u...
And Oil has already done it. It needs to be faster, but that's an inevitability based on progress so far.
Why Use Oil? http://www.oilshell.org/why.html
http://www.oilshell.org/release/latest/doc/simple-word-eval....
You can try it right now. Turn on 'shopt -s simple_word_eval' to get the better behavior.
Or turn on the group 'shopt -s oil:basic' and get even more better behavior.
When those options are off, Oil runs some of the biggest shell scripts in the world. It's the shell that's most compatible with bash by a mile.
https://github.com/oilshell/oil/wiki/Shell-Programs-That-Run...
String interpolation isn't done via quoting but via the format method on the string type, e.g. `"foo {} baz":format "bar"` generates the string "foo bar baz".
Similarly, to execute a command on a remote host, pass in a closure as an argument to the `remote:exec` command, like `remote:exec {1+1} "example.com"`, which will return the value 2.
I find that when quoting becomes a problem it usually means that the wrong solution has already been picked. If you're doing double quoting, you have already lost.
This is why I started my shell. Eventually it will have the ability to run both POSIX and other shell languages in the same program.
Assuming everything is a table is overly restrictive. Assuming everything is text is the UNIX way. It's then the job of the terminal to render it.
The power and usefulness of a given language is often the ability to serve multiple goals at once. Python as tool is cool specifically because of a large-ish but accessible-ish library covering a everything from regex-ex to a web-server.
This project might not serve the goal of writing high level programs and interacting effortless with the command line/system. But I don't one should automatically dismiss the goal of making stuff-glued-together look like a coherent whole.
And before you know it you have a buggy monster written in a batshit insane language that nobody is going to want to touch.
If your shell is at least vaguely sane, the situation is a lot better. I don't really get why you would argue against something that fixes the many many many mistakes of Bash.
I see many insane constructs that accidentally work. I also know how much time it would take to explain expansion rules. And people don't even want to hear it because it shakes their simplistic (and inconsistent) understanding.
So yes, I hope we'll see something like crush replacing Bash. But I don't see how it will happen.
U should fix ur computer rather then spreading FUD.
Also its clearly stated that copy of pwsh is intention on both projects.
Nushell is still one man show just like crush so they need decade or so to get to pwsh level.
Its nice to make stuff and offer alternatives but lets keep perspective here.
My computer runs just fine. Do you really think pwsh is fast?
Vim is slower for example but rarelly anybody compalain about it given its power.
(I'm not suggesting these aren't valuable - these are valuable experiments, but I'm saying for them to succeed, a new OS is needed.)
The native OS doesn't really embrace structured communication, and so commands will still have to treat the structured shell as a second-level citizen.
In an ideal world, for the structured shell to work, 'npm list' should return a table (or whatever the native data structure is) and you should be able to do 'npm list | where {deduped==false}'. In the real world, 'npm list' won't support the structured shell, so you would be writing 'npm list --json | jq '<obscure commands>' | json:from | where {deduped==false}'. And then 'npm list | grep -v 'deduped' is more simpler, so you don't really feel the advantages of the structured shell.
I think for this to be solved, the OS's primary communication system should be structured, so you can guarantee that every command will be a table. Then the programmers for the tools of that OS will output a table, and the users will be able to take advantage of this. Until then... it's basically a fun experiment.
You mean the STDOUT format of all processes? I disagree. `npm list --json` already does what you want. Your only argument is that ` | jq '<obscure commands>' | json:from | where {deduped==false}` is too obscure. The solution is (probably) a simplified `jq` that has the query format you want. Problem solved. Keep it simple.
As with haskell, it'll feel as though I've arrived somehow, should I achieve real understanding.
If you'll bother condescending to such, you'll find one of the most interesting languages I've encountered in years. jq's syntax and execution model is beautiful. It has some rough edges, leave no doubt. Some function names, date handling, and the obtuseness of some of the errors it returns can be frustrating. But its data streaming layers of nested closures are an extremely terse, readable, and powerful way of expressing operations over structured data.
I'd would suggest to spend that time on learning how to effectively learn new languages. There's more than one way, and I guarantee that afterwards you'll conquer any given "snowflake" in no time at all.
There are plenty of mysteries "out there" in the problem domain; you don't need them in your tools.
npm list --json | ({ json }) => json.somePackage.author.name
What if it allowed parsing of XML natively, allowing instant scrapers to be written in a single command?
curl www.example.com | ({ xml }) => xml.find('.importantInformation') > scraped_info.txt
[0] https://github.com/kellyjonbrazil/jc
(disclaimer: I am the author)
> What if it allowed parsing of XML natively, allowing instant scrapers to be written in a single command?
Powershell’s Invoke-RestMethod parses xml natively to a structured output; it’s useful but not enough to make XML fun or trivial to work with beyond root.node.foo.title simple scraping.
What I meant by this is that the newline character is the primary delimiter of the data, but I take your point. IMO, the best way to tackle the myriad ways of structuring data is to write parsers for them, not to rewrite the programs themselves to work with your shell's idea of structured data. That's a losing battle.
I think that’s a losing battle as well due to the brittle nature, any tweak to any command output could break something, meaning endless maintenance work on a large scale.
Rewriting concepts into new commands at least bring consistency and a chance of shrinking maintenance work as it settles on a nice design - and if not the maintenance won’t be endless rebuilds of the same parsers.
[0] with a hex editor it is possible to create a file system entry using '/' in its name. Linux does not handle this situation with grace.
[0] https://github.com/kellyjonbrazil/jc
(disclaimer: I am the author)
I'm not sure about similar xml cli tools, but they probably exist.
Point is, both of these should be cli tools, not shell built-ins.
I :heart: xmlstarlet (sometimes installed as that name, sometimes as just "xml"): https://formulae.brew.sh/formula/xmlstarlet#default => https://xmlstar.sourceforge.io/
The xsltproc that ships with libxslt1 is also pretty handy, although more verbose
While not quite xml, pup is super handy for doing quick html selections: https://formulae.brew.sh/formula/pup#default => https://github.com/EricChiang/pup#readme
xsltproc of course
OS system calls are highly structured, down to the argument position and data type.
Having something more than a blob is great.
Of course, there will be a performance tax.
The whole “But everything’s a text file!” is an outrageous lie, and always has been: just an excuse to cover up a massive omission in the design of the Unix file system (untyped, untagged “file” resources). Such egregious corner-cutting might have been understandable—even forgivable—back in K&R’s day, when they were bootstrapping C and Unix on hardware with less processing power than a modern microwave oven controller. But we’re decades on from that now and there’s no excuse for perpetuating it: we know how to build safe powerful modern static and dynamic type systems, and we’re spoiled rotten for the hardware resources to support it.
Fifty years ago, K&R built C and Unix, in a cave, with scraps. Nowadays you can bootstrap a complete hardware stack for less than the cost of a 360, and there are modern kernels (L4) and systems language (Rust) already available for free off the shelf. So pick a target market and build the userland for that. All that’s missing is a modern-day Kernighan and Ritchie to step up and get on with it, so it’d be a sorry indictment of this generation if they do not exist.
I think the problem then, and the problem today, is that these “types” are a very very mobile target. How many implementations of JSON are there? XML? HTML? CSV?
It doesn’t make sense to put these types in the operating system, or else you’ll start TypeError’s every time you download anything. Not to mention the Computational costs of parsing the entire structure of a document every time you edit it to verify that it’s “still of type X”.
I’m not at all sold on the idea that categorizing files and streams in this way is a step forward (in the shell, at least). This is what applications are for: to define their own formats, parsers, and generators. The point of the operating system is not to be one of these applications, but to facilitate their development and use. IMO, typed files and streams would actually impede more than enable this work.
The OS doesn’t need to hardcode individual types, and to do so would defeat growth. It only has to provide a standard mechanism by which files can declare the type of data they contain, and executables can declare the type(s) of data they consume and/or produce. BeOS, for instance, tagged files with MIME types. The original Mac OS used “four-char codes”. Even DOS had file name extensions, as awful as that particular design choice was.
Storing arbitrary data with no way to indicate how that data is encoded is rank insanity. Saying “it’s the app’s job to deal with encodings” is not wrong, but if apps have no way to know what encoding a given file was written in, how can they know how to decode it?
Please do not rationalize away the longstanding and well-known deficiencies of Unix. These deficiencies do not aid progress; they hinder it. As do apologetics.
I think you're conceding that the OS would not be responsible for enforcing data's compliance with it's declared type.
I'd be OK with this world, but I am not convinced that this hypothetical OS is substantially better than the UNIX descendants we have today.
> Storing arbitrary data with no way to indicate how that data is encoded is rank insanity.
I think we all can agree on this... it's just a matter of how we encode the data type. UNIX descendants follow the pattern of encoding data types with filename endings. While I agree that this approach feels primitive, it's hard to argue that it doesn't work, because it does. This is fundamentally identical to MIME-types, imo, because I believe extensions (and mime-types!) are fundamentally "names", and don't have much semantic above that, given the diversity of "name"-implementations, as stated earlier.
I don't believe that we gain a lot of practical value from compelling executables to register their accepted "names" with the Operating System: I'm trying to imagine what big value this gives you, beyond transforming `some_program --out_type=json | jq` to `some_program | jq`.
I hope you understand I'm not trying to apologize for UNIX. I agree that many things about it should be better. But I do think they got "everything is plaintext, from the OS's point of view" right.
(I think it's good discussion and I want to learn from it)
An ad-hoc arrangement with little formal support at the GUI level and none at all at the shell level.
`some_program --out_type=json | jq` to `some_program | jq`.
JSON only tells you how the data is encoded, not what it means. Contrast this MIME type:
application/atom+json
Knowing data is JSON-encoded only enables (unsafe) generic JSON operations. Knowing the data is, say, ATOM allows task-specific operations. Consider a search tool. Having precise type information enables format-specific parsing operations to be decoupled from generalized tree/table searching operations.
Not only that, but looking up parsers and inserting them into the pipeline can be completely automated away. `some_program` doesn’t need to be told to output JSON; it can negotiate that itself with the program that follows, the shell acting as broker. (Content negotiation is one of the Great Ideas of HTTP; sadly utterly ballsed by the WWW, but that doesn’t mean others couldn’t do it right.)
Or consider `ls`. How many options/arguments does that have? (If you don’t know off the top of your head, I totally understand and am happy to wait.) Now, how many out of those options relate to recursion, filtering, and presentation? Because none of those should be built-ins; they should all be separate, general, composable tools. How much duplication of effort would be eliminated across the entire Unix command line? How much easier would it be to learn the entire system?
This is Unix Philosophy 101, yet even a simple standard tool like `ls` utterly fails it. Why? Because for effective, efficient, safe composition to be the easiest route, it needs to Just Work. Which it can’t do, because lack of IO typing and tagging means that What You See Is All You Can Get. Consider: for some consumers it’d make more sense for `ls` to output a list of inodes; for others, a list of full paths; for others, URLs. Outputting a list of file names is crap (assumes shared global state—cwd—that will not change between time of creation and time of use). Outputting a list of file names arranged in multiple vertical visual “columns” each offset by N spaces is insane. This is the very definition of Big Ball Of Mud.
Like I’ve said, there is no point trying to “fix” Unix because fundamentally it does not want to be fixed; never mind whether or not fixing it is even technically feasible. But that’s fine: there’s value in stability and it is a sunk cost so nothing is lost. Just leave Old Unix to be itself, which is what it’s good at, and build a new system unencumbered by legacy baggage and baked-in mistakes, making full use of all the wisdom and resources gained over the decades since.
Okay, so a brand new system will have to be radically better than the old in order to attract audience, but that’s totally doable. Heck, there were already way better CLIs forty years ago (VMS, Plan9, Symbolics Lisp, Smalltalk), so it’s not like there isn’t a wealth of knowledge and experience to purloin for free. It’s just a question of motive: who now benefits from propping up all the old complexity and makework vs who’ll benefit by completely disrupting it. And you just have to look at how an aggressive reborn Apple handed the mighty untouchable Microsoft its own ass back in the 2000s to work out the answer to that. Or Kodak. Or Xerox. Or the US car industry. Or the British Empire. And so on.
Qui audet adipiscitur. She who dares, wins.
“jq 1.5 released, including new datetime, math, and regexp functions, try/catch syntax, array and object destructuring, a streaming parser, and a module system”
These big bloated behavioral silos that keep popping up on Unix are neither good nor accidental: they’re the symptom of a fundamental fault in the underlying host. Because when the most complex concept it is capable of communicating on clients’ behalf is “Ugg bang rock!”, it should come as no surprise when its clients all go “fuckit, I’ll just do it all myself”. And so it goes.
--
I wonder if instead of an OS solution we can just add more metadata to the actual binary files we compile. For example, if we have a new ELF format that has an optional list of commands, options, full input/output types, etc, any one of the shells can start to provide better tooling. Output format types can then be passed along with the stream, and potentially checked that they work as the proper inputs for another command. Let the shell auto generate the simple cli help docs, or generate the bash completions, or figure out how to better convert data to desired types.
STDOUT_ACCEPT=application/json,application/csv,text/plain
The other trick is knowing what an application will accept via STDIN - ideally at runtime, but before actually writing anything.
If the shell understands all this, it could automatically format JSON/csv output for display, integrate jq for formatting, or even convert between formats (eg: when one app writes in yaml, the other understands JSON input).
1) I don't want shell to be full-blown high level programming language. POSIX, Bash scripts/languages are enough for my shell uses.
2) For anything high level or for tasks shell would become complex, We do have dozens of fully matured languages. I would rather use those instead of this/any shell script.
3) Features desirable for shell include syntax highlighting, autosuggestions,command-not-found helper,tab autocomplete, flag complete. My shell already has it.
4) I like bash and zsh as shell. Bash is simple,i like it,but it doesn't have few features that i mentioned. So,I don't use bash as my main shell. zsh also is compatible with most bash/POSIX and it has all tte features that i mentioned in (3).fish also has these features but it's syntax is different which i don't tend to use. So,i don't use fish. zsh has all features that i like in fish too(mentioned in (3)) .
And that's okay. Personally, I'm thrilled by the idea of a new shell. I don't plan to use Crush in production (yet?!) but I definitely plan on installing it on my laptop.
I'm excited to see how things go. I understand, for a while, it won't have all the bells and whistles of zsh. But that's okay. Everything has to start somewhere. I remember back when alacritty didn't even have easy scrollback!
Best of luck to Crosh and its author, liljencrantz!
Crosh is the ChromeOS shell!
It's a supercharged version of bash that works with Windows, but instead of piping character-only data, it can pipe structured data between commands in the form of objects. The reason I ask is because, while it adds a ton of flexibility to shell scripting (to the point where I consider it to be a language), it is also quite complex.
The benefit of shell scripting is that it is simple enough to have as a tertiary language that you can pickup and use after a six month hiatus. Once you get into more powerful language territory, there are a lot more components to memorize and it becomes difficult (for me at least) to keep those idioms in my head if they go unused for months.
I definitely did not find that to be the case. I've learned and relearned shell scripting a bunch of times. I always forget it and need to relearn it. Worse than perl.
I've been using/abusing traditional shell scripting languages for over a decade now and I still struggle with the rough edges (especially quoting/escaping/argument passing).
While it was neat for interactive use to have stuff like proper string manipulation and list comprehensions, writing scripts with it (.xsh files) was horrible, because the sorta-but-not-really-Python-ness meant that no tooling worked. Syntax highlighting was messed up, code autoformatters barfed, linters barfed, autocompletion didn't work.
I actually find writing scripts in Bash, aided by the amazing linter https://www.shellcheck.net/ to feel almost like it's now a compiled language. It goes from being a fraught endeavor, to being kinda fun and educational. It's like pair programming with someone who has an encyclopedic knowledge of the language quirks. Hook it up to automatically run on-save in your editor!
from marcel.api import *
for file, size in (ls('marcel', 'test',
file=True, recursive=True) |
map(lambda f: (f.path, f.size))):
print(f'{file}: {size}')This would actually be possible to fix, but it's quite a bit of work.
I also feel Crush addresses some issues I had with Fish. Still on Zsh though (I like pastable scripts).
cat urls.txt | mario async-map 'await asks.get ! x.json()' apply len
Nushell also seems cool, it's somewhat similar to Powershell. https://github.com/nushell/nushell
This is not correct. Powershell allows calling into COM, .NET or straight DLLs for that matter.
It is also what I see as missing opportunity in most UNIX shells that failed to build up on Xerox PARC, Lisp Machines and ETHZ experiments.
The ability to directly plug into shared libraries, UNIX IPC, and modern mechanisms like DBUS and gRPC, alongside live debugging capabilities, thus exposing the whole OS stack to the CLI, making it a proper REPL experience.
Other than that, it looks quite cool.
There should simple be a grpc command that you use to instantiate a connection to a grpc server with, and that lets you do discovery, send commands, etc.
It's on my todo list for crush to do that for grpc and dbus.
The ability to run existing POSIX and bash scripts is the biggest difference between Oil and other new shells (something I'm often asked). Unfortunately that's also why it's such a long implementation effort. But in reading the comments here, I still think that was the right choice.
----
FWIW Oil's approach to structured data is to use interchange formats over pipes. You can already pipe JSON, HTML, and CSV/TSV over pipes, and people do. There are many command line toolkits that do this.
https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil
However the shell is also missing few things which would help that approach, so I'd like to build some native understanding of those formats into it, without requiring any special "plugin interface". Plugins are simply processes in Oil.
For example, Oil already has a QSN "Quoted String Notation" to properly escape byte strings like filenames (this happens to be the format of Rust string literals, which are similar to C and JSON string literals). So you don't have to do parsing and splitting.
The approach to tables will likely be built on QSN in TSV or QSV. And then you can enhance "npm list" and "dpkg" just to print that to stdout, without having to integrate into any special "plugin system". It just works with normal Unix processes and the kernel.
https://github.com/oilshell/oil/wiki/TSV2-Proposal (not implemented)
http://www.oilshell.org/why.html
Let me know what you think! (feedback link in the article)
ls -l /usr/bin | jc --ls | jq '.[] | select(.size > 50000000)'-rwxr-xr-x 1 root root 51859776 Jan 26 2018 pandoc
IMO people have been trying to do the same thing with these tools since they were invented. Better to learn how to use them the way they were intended before reinventing them.
Edit: re-read my comment. Seems a little negative. Sorry. To be clear, I think the jc/jq approach is much better than the roll-your-own parser in shell approach.
SQL is a neat tool for what it is. But command line bashing does not seem like a good fit to me. I get what you are saying though. I just would be kind of miffed to have to do something like 'select * from folders(/foldera/folderb/folderc) where lastcreatedate > getutcdate('1970') order by lastcreatedate desc' when I can currently do something like 'ls -ltar'. It is not that I can't do that. But it takes a cognitive load to do it for a command I use a lot. Of course I could make a program to do it. But then I am kind of back where I started? I could get behind SQL in addition to what I have. But in replacement I think that would be a tough sell.
And a proper shell, Closh: https://github.com/dundalek/closh
This is repeated ad infunum by the anti posh crew, while good arguments are presented against such attitude all the time.
Also, xargs is simply awful compared to what posh gives and awful without any comparrison.
https://github.com/jhspetersson/fselect
I've integrated this into my workflow, it's a polished tool.
I get that many Unix commands are obscure, and the syntax is often inconsistent, but once you're familiar with them, you can do everything this shell does and a whole lot more. And for anything more complex you've got a million other languages to choose from.
The rest of your command is subjective, but I find that being able to see e.g. what user has used a lot of CPU and has many processes running via the command `ps | group ^user cpu={sum ^cpu} proc_count={count}` is both useful and far harder to do in a traditional shell. YMMV.
Like it or not, the Unix pipe model of command integration is what enables programs to "know" about each other without ever really knowing about each other. Crush looks like its foregoing that.
- Easy process execution, including branching on exit codes, overriding env vars, providing a working directory, etc. This must support concurrent processes (e.g. pipelines).
- Easy file and stdio management (i.e. piping). This must support streaming (passing along chunks of data as they arrive, rather than gathering them into big intermediate values).
Scripting language REPLs provide everything else I care about (string handling, looping, variables, etc.). Their built-in support for these two things is usually really bad, but libraries can plug that gap. I've tried a few for Python, Haskell, Scala, etc. but the best experience I had was Racket's shell-pipeline library ( https://docs.racket-lang.org/shell-pipeline it's also used by the Rash shell that I've not used https://docs.racket-lang.org/rash ). This makes really good use of Racket features, e.g.
- Quasiquoting for mixing commands with normal functions in a simple, sane way
- S-expression syntax works well for shell-like code, e.g. no comma-separation, distinguishing between (unquoted) symbols for names, flags, etc. and quoted strings for data, allowing special characters in names (e.g. to allow things like > in our pipeline), etc. Of course, Racket can be written with I-expressions, sweet-expressions, etc. too, but I've never bothered.
- Stdio redirection works with Racket's existing "ports" mechanism
- Env vars can be managed using Racket "parameters" (dynamically-scoped variables)
(Note that the API seems to have changed a bit since the version I used, so I'm not sure how the new stuff like laziness fits in)
Trying to create a combined shell/programming language is a mistake, in my opinion. Marcel exposes bits of python on the command line, e.g. for writing predicates. For more serious programming, marcel offers a Python API that allows the use of shell commands inside Python, neatly integrated into the language. For example you can write a series of commands, connected by pipes, and the result defines a Python iterator so that you can use the pipeline results in a for loop.
See the website for more details and discussions.
I love the idea behind PowerShell, but the verbosity turns me off.
PowerShell, of course, has the pipelines but also COM integration, which is a big deal for Windows (you can script "the desktop" via COM; for instance you could create a new instance of Microsoft Word and call objects inside Word to do something to a document.)
Unix has dbus and some similar things, but nothing quite so critical as COM is to Windows.
Did I mention it also makes readings scripts at least 100 times easier?
ls *.txt | awk '{print "mv" $1 "/somewhere/"}' | bash
Maybe the closest one gets of it are GNOME, XFCE and KDE based distributions.
Although broading the view a bit, macOS has XPC/Open Scripting Architecture and ChromeOS/Android/Fuschia have their Binder infrastructure.
Currently there is no documentation beyond the introductory GitHub page. But this isn't a fork of sh that only needs documentation of the minor ways in which it departs from POSIX compliance: there is an unknown set of internal commands, each with a complex table format for its output (it is necessary to know all relevant column names to do anything), which depends on a large set of non-text types, which have complex object-oriented behaviour, with "serialization" to a variety of file formats, with an unspecified script and command line syntax.
Such complex features, on par with serious frameworks in full-fledged programming languages, need to be defined and documented very thoroughly, or nobody will understand, trust and use them. Unfortunately, there are many hints that not only documentation is vastly behind schedule, but that important aspects and practical considerations haven't been addressed.
For example: how can I quote file names containing nasty characters? How can I use environment variables? What is my home directory? Why is executing an executable program difficult, with complex and unclear mechanisms that mess with the command line? Given a very simple and very portable traditional shell script (e.g. ps -eaf | grep foo | sort) how can I do the same in Crush with minimal effort? If I load a very large file, does the shell choke attempting to load it all to memory, or does it process it in a proper streaming fashion?
Other issues are more fundamental. How can I perform relational queries between multiple tables, e.g. a join between ls output and a file listing users? If I can't, what can I do with tables that cannot be done more easily and flexibly with awk, jq, and other established tools?
Theres nothing wrong with that, as long as you don't create unrealistic expectations.
A new shell must be similar to existing ones for compatibility with an enormous ecosystem of command line tools that work well in a typical shell; it must do almost everything that traditional shells do equally well (this includes documentation; documentation is non-negotiable) but it must also have some compelling advantage.
The right way to collaborate on designing a shell is to discuss possible designs of command syntax and behaviour, then if some promising idea comes out of the brainstorming refine it to a full specification, then if the idea survives (without mutating into a less ambitious scripting language, or a library for an existing language or shell, or a small change to an existing shell) begin an implementation. Starting with a proof of concept without a well developed and well documented design is the wrong way to offer a "product" but also the wrong way to learn.
Assuming the answer is nowhere:
Please retract. You appear to be one of those people with no understanding of modern open source culture, and only the ability to criticize. Do you expect everyone to keep their projects private until they meet your personal standards? (a) why the fuck should they? (b) what about a Github user who simply lacks the skills to ever meet your standards? (Evidently not the case here.) It's a shame I'm being overtly aggressive, but I think it's commensurate with the less overt unpleasantness of your comment.
I bet you're one of those people who thinks it's "irresponsible" to publish software on Github that could have bad consequences if someone were to rely on it in an important context. Repositories such as Github are free. They are a place for anyone to store whatever creations they come up with. It's a huge and uncuratable mixture of essential backbones of modern computing, art, utter incompetence, and technical genius, and a lot of stuff in the middle. How that is hard for people like you to understand I do not know. I guess you have a very boring job at a very boring company.
I just wanna write a few small lines of a shell script which downloads stuff;
It is very very stupid through what loops you hvae to jump to do this with curl and bash script in a good way including readability and error handling.
Is a perl script really better?
Perl 5,6 and 7 did not make it very sane to me, managing and installing and working with perl is also less trivial then having a shell like bash, sh or dash available to you.
Something inbetween bash and go is missing, i like the approach of crush. I think thats the biggest misstake happening in bash: playing around with strings and auto expansion and substition etc. Its weird, its error prone, it doesn't have a proper clean well defined ecosysem.
You don't lose much in terms of shell interop though - the native exec command understands a lot of the basics (file redirection, pipelining, background execution). In the simplest case, it's just one more word:
exec wget $URL
# Dead
error "IO error"
# Not dead
if {catch [error "IO error"] error_or_result} {
# The error
puts "Caught: $error_or_result"
} else {
# The result
puts "OK: $error_or_result"
}Plus the fact that PowerShell is .NET-based and not the most snappy shell out there either.
Plus the fact that extending something requires investing a lot of work just understanding how things tie together, and then avoiding messing up all the hacks and idiosyncrasies added along the way instead of having a clean slate to build on.
And if it can't do that because of a reliance on shared libs, then that's even worse!
crush> ls | where {type == "directory"}
ls -l | grep '^d'ls -l | grep '^d' | awk '{print $9}' | xargs realpath
Or
ls -ld "$PWD"/* | grep '^d' | awk '{print $9}'
Guessing Crush could just add another property called Fullpath you can echo in the next pipe after filtering.
ls | where {type == "directory"} | select %Fullpath
Maybe he adds a --dir flag
ls --dir | select %Fullpath
How long do these tools exist? Over 20 years?
What's always bothered me about piping to grep is losing the original syntax highlighting. Particularly annoying when the first command, whose highlighting I want to preserve, is itself grep; this leads to awkward contortions like
git grep "a" | grep -v "b" | grep "a" find . -type dcurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
command also which upgraded by Rustc to rustc 1.45.2 (d3fb005a3 2020-07-31).
This is one problem I have faced with Rust quite frequently. Try to clone and run a random (not famous) project from Github and it doesn't compile.
Love Java when it comes to this. It's easy to see which version of the JDK is needed from the Maven file. If you have that version of JDK, the projects almost always run.