Introducing nushell
jonathanturner.org
jonathanturner.org
What if, instead, we pushed the approach “upstream”, asking systems to have an additional stream that all shells can access? We have stdout and stderr, we could add “stddata” - a pipe that expects and produces structured data. Then it would become trivial to add support around the userland, without displeasing anyone - because it’s just an additional interface, not a replacement. The pipe should support two or three formats to start with (csv, json, and maybe xml, so you can handle both tabular and child-parent relations with a varying degree of precision) and shells could have a special character for piping stddata (I like ¥) so everything else about them would stay the same.
Probably it might benefit some. Good to see competition in this area with sh, bash, zsh, fish, conch and many others.
Ever tried to rewrite one (and the software on top of it, since we're "including" the OS) :)?
I admire your optimism and I, too, wish we'd leave this POSIX hellhole behind us (and I'm fond of this POSIX hellhole!) but I think at this stage in the story of computers, it's no longer realistic.
If you ask me, no, but that's because I remember (and not very fondly) an age when it seemed like we'd have a new arbitrary code execution bug in a PNG or JPEG or PDF decoder every week. I don't really wanna go through it either.
Edit: Lots of people don't remember (or don't know about) that period and think writing an OS and all the boring stuff that goes with it is easy. Four or five years and it should be all done, eh?
That's also why we got things like the security community, which is full of enthusiastic people and big companies that know exactly what OpenSSL needs and how it should be written. But when it comes to sitting down and doing it, or committing the money, the crowd gets awfully small. Foundational software isn't too glamorous, and writing things like a PNG decoder is pretty boring, too.
I know a lot of it like the back of my hand, I've been writing POSIX code (or, at one point, POSIX implementations) for much, if not most of my career. That doesn't mean I don't want to see something better, or that I'm blind at the fact that Plan 9 is so fun to code for precisely because they skipped POSIX (although, that being said, I'm also not blind to the fact that it's on of the big reasons why it never really took off, either).
Another option would be to just use the 4th fd.
After all, /dev/std{in,out,err} are more or less just convenience links (allowed extensions of POSIX). But using /dev/stderr might fail on some POSIX compliant systems.
Personally i prefer the keep it simple approach. Id much rather /etc/hosts stayed the way it is rather than becoming xml, json or binary.
The whole idea of "Power" shell is broken if you ask me. Shell should be simple. If it isnt, write some proper code. And do some code review, and testing first!
I honestly don’t know how much work it would entail from OS devs, but once you have it in Linux and BSDs/Mac that’s basically it. Microsoft could take the chance to be an early adopter, considering they already have all the machinery to output structured data in PS; at that point, whether it’s pure posix or not would matter little in practice.
I don’t particularly like PS for a number of reasons, but I do admit that its output facilities can be quite nice. Having something like that in Bash for the few times one really needs it, with standard interfaces and without having to learn a completely new paradigm, would be a nice win imho.
This doesn’t mean we should touch /etc/hosts or anything, btw.
exec 3<> somefile.txt
It also means shell command output can be easily transposed into JSON and CSV. Actually pretty clever!
That said, since I'm a Perl developer, I'll generally just go straight to perl's inlined implicit loop mode (-p or -n) if I need to use more than a couple of those utils, unless I want it to be portable with a minimum of requirements (as even though more binaries are used in the shell version, they're all essentially guaranteed to exist on a Linux/Unix).
Would an ideal situation be “do + while”
Whereas one may write
While x is true do “blah blah” && if output is still true then end elif do &&?
Or is that basically how it already works and we just have shitty interactions with the machines to get them too do the above (unless youre a more seniorbash person ((its one of my career regrets that i didnt spend a ton of time bash scripting)))
I enjoyed converting our many build scripts into Powershell, but using it as a shell was just uncomfortable.
It's not clear to me why. If I had to guess, besides the verbosity, it was the additional "indirection". Nothing was what it appeared on the screen. It was something else.
Bash felt far more comfortable as a daily shell because what you see is what you get.
I want to try Nu because it appears to combine that WYSIWYG simplicity of bash with the Powershell quality of passing data rather than text. Maybe tables are indeed a better data structure to use than objects.
For regular shell usage it's a bit painful because the abstractions are a bit too far removed from the file system you're operating on. But for scripting, such as managing and configuring systems, it's remarkably fluid.
The "nothing is as it appears" problem never bothered me much because it's still just data. You just specify the output when you're done crunching it. IMO being able to operate with typed data is really nice because you don't have to worry (nearly as much) about serialization between commands.
Scripting is programming, so you want data structures. For interaction, you want human-readable syntax.
e.g. programming languages mostly have a syntax: C, python, bash, fortran (exception: lisp), but they operate on data structures.
We can see powershell, nushell and other structured shells (e.g. some use XML) as an attempt to "get out of the syntax business", which XML began and JSON finished (https://blog.mlab.com/2011/03/why-is-json-so-popular-develop...).
This has been taken further into human read/write domains... with mixed reception: ant, XSLT, json as a config format. After all, parsing and "the syntax business" has a practical purpose.
The key to success in using PowerShell as an interactive shell is to memorize aliases and not caring about casing and spelling everything out. PowerShell is case-insensitive, has aliases (e.g. ? instead of where-object, or % foo to get the foo property of every object) and will accept abbreviated command and parameter names if the abbreviation is unambiguous. It is also useful to know about the different formatting cmdlets: mostly format-table (ft), format-list (fl), and that you can pick properties with -p (-Property). While PowerShell usually does the right thing, with these you can get the most appropriate formatting for any use case.
I commented on this earlier but the defectiveness of PS as a shell hit me almost immediately when I tried to recreate my workflow of 'dirs -v', pushd and popd with a large number of paths on the stack.
Being able to dump structured information out of an exception directly into a log, and then from a log into a database, without any loss of information or extraneous log parsing, is a clear win. Or from a command as simple as listing files ("ls") into a database or into any other tool or program. Outputing only line-oriented strings is just throwing away type information, and creates more work for everyone else, even more so than continuing to stay with lines processing tools.
"Worse is better" isn't for everyone.
"Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface."
Perhaps I'm wrong, isn't nushell simply adding one more dimension?
Instead of a one-dimensional array of lines, the standard is now... a two-dimensional array of tabular data. Perhaps it is not strictly a "text stream" but this does not seem to me to be a violation of the spirit of the message.
Simple line-based text streams are clearly inadequate IMO. 99.99% of all Unix programs output multiple data fields on one line, and to do anything useful at all with them in a programmatic fashion you wind up needing to hack together some way of parsing out those multiple bits of data on every line.
maybe check out Powershell
It seems like overkill for most/all things I'd ever want to do. Powershell objects seem very very powerful, but they seemed like too much.
Nushell seems like a nice compromise. Avoids the overkill functionality of Powershell.
Just thinking about a way that perhaps an old tool can be “wrapped” to be tricked into working with 2+-dimensional data by somehow divvying up the 2+ dimension input data into concurrent 1-dimensional streams, but this seems to require a way to represent more than 1 dimension of data without breaking existing utilities (unless there was, like, a wrapper/unwrapper layer that handled this...)
There's no real alternative for many profesionals, if they want to be employable.
Maybe we should accept this fact and try to make everyone's life easier than be stuck up and push people away.
There's no law that says that Unix tools can't be extended.
And heck, basic POSIX tools violate so many Unix principles...
I don't write complex scripts in shell anymore because it's insanity. But ad-hoc loops and crap like that... hell yeah. At least a few a day. Sometimes dozens.
People need to be reminded, I think, that shell isn't a programming language first. It's a user interface. And when I look at Powershell scripts and other things of that nature and think about living in Powershell day in and day out I don't see the big pay-off over something like fish.
'Is this going to make it so I can get my work done faster?'
'Is this going to be more pleasant to use as a primary interface for a OS?'
When I go into fish or zsh and use curl to grab json and 'jq' to get a idea of how to use the API in python or node...
versus running 'curl' in powershell against some random API I have never used before..
I get the distinct impression that 'This is fucking garbage' in that It would take me a lot longer to figure out how to use powershell in a useful way then the time I would save by doing so in the long run.
Unix follows the KISS principle, and that is key for success. Albert Einstein said: "Keep things as simple as possible but not too simple". In that sense Unix and Posix are well done. However, that doesn't mean that good ideas like Nushell are not welcome.
[0] https://github.com/nickcox/cd-extras#navigate-even-faster
I'm trying to elicit more of a response from the comment author. I think they make some good points. I would like to learn more about their ideal system.
Or even something like --format-rfc4627 filename,size,permissions-numeric to get only those fields out in a valid JSON format.
This wouldn't remove the "every program has to know how to parse the output of every other program", but I am not convinced it is needed. For instance, how would e.g. grep know what field you want to process? And does e.g. "size" carry universally the same meaning and semantics in all programs there is and can ever be? Ditto for "permissions". And what about "blacklist".
As a completely fictious toy example:
(ls --format-rfc4627 filename,size,permissions-numeric | json-shunt -p "filename" --cmd grep "MegaCorp") | json-print -p filename,size
Yes, I know there are more concise and elegant ways to express the same thing of printing out file sizes and filenames of matching filenames... Also, when scripting pipelines the verbosity doesn't matter, IMO it'd actually be a benefit to be able to read the script.
Edit: fix pre-caffeine wonky redirection and brain fart
The trouble is, comments are normally defined as having no effect, not part of the data model. But iff the comments contain useful information, how do you pick it out using the next tool? Imagine having to extend `jq` with operators to select "comment on line before key Foo" etc... And if it is extractable, are they still comments?
How do you even preserve comments from input to output, in tools like grep, sort, etc.
XML did make comments part of its "dataset". That is, conforming parsers expose them as part of the data. Similarly for whitespace (though it's only meaningful in rare cases like pre tag but that's up to the tool interpreting data), and some other things that "should not matter" like abbreviated prefixes used for XML namespaces). This does allow round-tripping, but complicates all tools, most importantly by disallowing assumptions that some aspect never matters, e.g. that's it's safe re-indent.
I'd argue the depest reason JSON won over XML is that XML's dataset was so damn complicated.
Doug McIlroy (2003). The Art of Unix Programming: Basics of the Unix Philosophy
That said, I’m looking forward to testing our nushell!
not so sure about your assertion there eugene :o)
interfacing with funk formatted lines of text is waay more easier because (imho) of the 'wysiayg' principle, and encourages, for lack of a better term, 'tinkering'
> Settling on a common data simple/universal format robust enough for all purposes...
output from 'ls, gcc, yacc/bison, lex/flex, grep, ...' should all follow the same fmt, sure...the good thing about standards is that we don't have enough of them to choose from, f.e. xml, sgml, s-expressions, json, csv, yaml, ...
having said that, when dealing with copious volumes of data, structure doesn't hurt, but in such cases, my guess is, there are only a handful of data-producers, and data-consumers for automatic consumption, and are all tightly controlled.
However, the unstructuring of data and keeping a single form of communication is what has kept UNIX around for so long and why Linux hasn't collapsed under its own weight long ago.
I was voicing your opinions exactly when I was new to the professional computing world. Over time I saw a lot of structured data schemes come and go and they all fall down with this: inflexibility and improper implementations.
Where do you think does the structure inside the data come from? You need to come up with a standard. Now you need everyone to stick to that standard. You need to parse the entire data or at least most of it to get to the information you need, finding new security footguns on the way. Soon you will realise you need to extend the standard. Now you have not only conflicting implementations (because noone ever gets them completely right) but conflicting versions.
And this needs to be done for every single fucking little filter and every single language they are written in.
Take a look at the various implementations of DER and ASN.1. The standard seems simple at first glance, but I haven't seen a single implementation that wasn't wrong, incomplete or buggy. Most of them are all of that. And DER is a very old standard that people should have understood in the meantime.
In order to get at least a bit of sanity in all of this, you need a central source of The Truth wrt. your standard. Who is that going to be for Linux and the BSDs and for macOS? Linus? Microsoft? Apple? The ISO standards board? You?
And all of this is equally true for log data.
I'm okay with tossing backwards compatibility over board. But not in favour of the horrible nightmare some people here seem to propose.
In my limited experience, you're either taking the cognitive hit to learn how to grapple with line-oriented text wrangling, or taking the cognitive hit to learn how to grapple with line-oriented text wrangling AND on top of that a query language to fish out what you want from the structure first before passing it to the text wrangling. I'd sure like a general solution though, if you have a way out of this thicket.
For example, to sort filenames by length
$ ls | mario apply 'sorted(x, key=len)' chain
doc
src
bench
extra
AUTHORS
LICENSEIn their examples it looks like 'ls' is built-in to their shell instead of from e.g. coreutils.
That seems like a real mistake, rather than simply having nushell's "ls" be a wrapper for coreutils "ls -la" or some such.
I understand the benefit of reimplementing everything from scratch, as that way you have a more consistent nushell experience everywhere, regardless of which version of "native" ls your system has. And allowing "^ls" is a useful and necessary hedge.
But, wow reimplementing nearly everything seems like an enormous undertaking.
(It is of course possible that I'm completely misunderstanding things! Perhaps the devs can comment?)
There are definitely cases when you're better off with producer-side filtering. For example, `find` allows you to not traverse certain directories with -prune, which might save a lot of resources.
Cmdlets which get data, like Get-AdUser or Get-MessageTrackingLog in Exchange, or Get-CimInstance often have a lot of common filtering parameters as well as a more general -Filter or -Query "build your own".
Big source of annoyance where you can't do that, but might want to.
The UNIX idea of piping text output from one command to the next was fine at the time, but it means that the downstream command has to parse text. With nushell, you are parsing slightly more structured text.
The two steps they could have taken in addition:
1. Instead of generating rows of a table, why not lists of strings? E.g. ("abc", "def", 123) instead of "abc | def | 123". Much easier to deal with in the command receiving this input.
2. And instead of just lists, why not objects? E.g. a process object, a file object, or, if you'd like a list as above.
I have implemented this idea: https://github.com/geophile/osh. For example, if you want to find processes in state 'S' with more than 3 descendents, and print the pid and command line of each, sorted by pid:
osh ps ^ select 'p: p.state == "S" and len(p.descendents) > 3' ^ f 'p: (p.pid, p.commandline)' ^ sort $
'where' in pwsh / nush literally looks for keys with particular values.
That would awksome.
https://github.com/dkogan/vnlog#powershell-style-filtering-o...
Some extra ideas:
- Tables are amazing. But I think we need at least support for this "shapes" of data: List, Tables, Trees and maybe graphs
- I think will be nice to have an equivalent of Request/responses alike http. And then a way to introspect what a command do, like auto-swagger.
- Having widgets like table-browser, graphs, etc AS COMPONENTS.
In FoxPro you could say "browse" and it display a DataTable grid with full editing capabilities on the current table (or it ask for a table to open). In this case:
ls | ui_browse
This mean we could get a jupyter-like experience here!
- Some way to do curryng and maybe environment alias: You could setup an environment for the system, the user, this directory. In the environment is a .toml with vars and stuff.
The other thing where I think shells fail... is that are still old-style terminal. Having text output is cool, but having more rich widget system will unlock better the potential
Like when a http response say is json, text, csv, etc. This information unlock certain capabilities and will allow to provide efficient execution of queries.
If psql (postgresql terminal cli) declare their data is in the "tables" shape then it could take the |where name = "" query itself and execute it instead of output all the rows...
You might find DomTerm (http://domterm.org) interesting. It combines solid traditional-terminal support (including solid xterm compatibility) with a data model based on HTML/DOM. So you can "print" images, HTML tables, and more. DomTerm "understands" shell/repl commands as consisting of prompts, inputs, and outputs. You can add markers to group the output into logical groups, so it is automatically re-formatted on window re-size. And more.
Everyone has to reinvent the same basic tool output parsing because ls (or top or ps or...) can't spit out real structured data.
I'm sorry, I couldn't help myself. Your comment reminds me of an anecdote I heard from the early times of structured programming. When structured programming was just gaining its feet, there was a certain class of programmers who just could not understand why people would want to write structured code. You can do everything in assembly they said, you have much more control over performance, etc. They looked down on structured programming as not "real programming".
There's a lot of benefits to adding some structure to text. I don't think that Nushell's approach is the best one, but to say that there are no problems and we shouldn't look to improve things is just backwards. We should always look to improve our tools and our craft, otherwise we would still be stuck writing assembly.
Also, I'm not sure what your argument is, except "I don't get it"? It's fine, lots of people don't get stuff. Just move on, and maybe in a few years when it's mature, you'll see it again and go "Ah!"
There are benefits, true, but there are also potentially serious drawbacks. Chief among them, I would hazard, is the risk that we get locked into a format that didn't anticipate a (completely unknown) future need and we have to go back and rewrite everything again.
The beauty of text's lack of structure is that people are free to interpret it any way they please.
True, but you are adding something. And if you add things you don't really need, it becomes plain old cruft.
Like the others above, I understand the urge to make some things simpler by adding complexity, but for the long haul I'm convinced it's better to keep tools simple and add any needed complexity to the code you're writing (whether it's shell scripts or anything else). And then document the complexity so future you or some poor stranger can understand why the hell you did that in the first place.
But I don't look forward to reading comments like "X was deprecated so we're converting these tables back to text streams so they can be converted back to Y properly."
Aren't we now locked into a format (plain text) which is now becoming more and more of a pain (what type is this? is this text related to this other text?) and we're now having to go back to old utilities and add --json?
So I end up having to put everything in one giant call to 'find', which defeats the compositionality of shells.
Default posix/bash splitting of output by spaces is very big-prone and having to type "$array[@]" all over the place is annoying. But splitting by lines is much saner. Now newlines in file names, that's just evil and should be outlawed at OS level. I think OSX did this (?)
In bash life can be easier by setting IFS (Google "bash strict mode" though I have reservations about the IFS part).
But really, its easy to fix in a non-posix shell. Fish splits by lines and interpolates variables as arrays by default.
In essence, yes. Yes you can get pretty far with sed and the likes. But it's not because it's all we really need, that other things (objects in this case) don't offer an improvement and make things better/easier/more convenient. Just like we theoretically could get along fine with just the CLI, doesn't mean that it is the single best thing in the world for all possible occasions.
EDIT: wording.
I’m gonna give nushell a chance, because tabular textual data seems like it still fits the bill while also making it less awk’ward to massage the data as needed for the pipeline.
> [...] you just have to spend a little bit of time learning them
I can still remember the many times I tried to learn shell scripting and gave up too quickly. So 'little bit of time' might be an understatement. Actually, you need a good use-case, sufficient motivation and enough time at hand to bite through the first problems...
And show me a serious, modern programming language that is so susceptible to whitespace usage. Yes, there are some that care about whitespace (e.g. Python), but rarely it will cause a syntax error to not place a space before a bracket.
Another problem is that those text formats are a bit brittle. I am talking about having to use IFS="" and ls giving back something different to humans than to pipes. Most of the time you can solve it one way or the other but having to search frequently for solutions to such problems just sucks.
I am not saying that the old tools aren't worth learning. In fact, I love writing shell scripts (for the good parts) and can't remember the last day I haven't used a shell. But saying there are no problems to be solved is just wrong. So I think a little discussion about what could be done and some prototyping shouldn't hurt anybody.
The other comments about adding a "stddata" and handing this responsibility to userspace is a good one.
You shouldn't need to know a specialized DSL for each tiny core util.
However, its interesting to see so many dependencies that are required to build it (73 crates last time I checked) and as shells have always been portable across many operating systems, I wonder how portable nushell would be since it wishes to replace my current shell.
That's a bit too many. But it's not often you compile your shell from scratch.
I’m always impressed when people show up with fixes like this (that I wouldn’t even know to look for).
Just had a quick play ... on arch linux `yay nushell-git` already works!!!
One area where I got a bit stuck was around help. `man where` gave me nothing, `help where` also. I tries stuff like `ls | where type = File` and got a type error. I think it would be amazing if this thing onboarded people a bit nicer, "where needs a condition, to learn about where type "help where" ... stuff like that
Overall really enjoying the ideas here and I am absolutely going to be following this!
In a perfect world, a language's ecosystem would have a tiny stdlib and allow importing of libraries for everything else. The Linux philosophy would be followed strictly.
The problem is just that the overhead in securing, maintaining, and organizing all those libraries is pretty large, as we've seen by npm's repeated failures. Of course the *nix community seems to have largely solved the problem, but there's also a massive culture of volunteerism benefiting them.
- torvalds
For a FOSS language, a large stdlib can be similar to a curated repo in practice.
I'd imagine a feasible solution being a permission-based system (which might already exist), where programs and their dependencies do not have access to certain facilities, like the filesystem, other processes or the network, without them being declared in a manifest file of sorts. Permissions should be inheritable when explicitly specified, and a sub-dependency could not gain a permission it didn't inherit from its parent. Unfortunately this does not work so well with a shell, since the shell needs the ability to spawn arbitrary processes. At least the shell itself would not have network access, forcing it to rely on tools already installed in the system
Also, if we resort to some magical wishful thinking, then all the tools in the system follow this permission model and the package manager is aware of the permissions. You could then query the package manager for all installed software with a certain capability, like network access, disabling tools like curl and netcat.
The problem is that Rust has two issues:
1) In Rust, you don't pay for what you don't use. This means that a lot of stuff tends to be an external library since not using something means you can exclude it COMPLETELY. The issue is that things tend to get atomized more finely than most languages would do.
2) Rust is trying to not bake things into a standard library too early. This is a good comment about standard libraries from Python:
"The standard libraries constitute one of Python’s great features, but they tend to suck up all the oxygen. Programmers are reluctant to write libraries that duplicate their functions, so poor libraries in the standard set persist. Only a few people, like Reitz, are willing to write modules that compete with the standards."
https://leancrew.com/all-this/2012/04/where-modules-go-to-di...
Most shells work on just one of either Unix or Windows; the blog post specifically mentions Windows support as being something on the radar of the developers, which Rust is arguably a better fit for than C/C++ (which most shells are written in) due there being a single compiler, standard library, and build tool with first-class support for both Unix and Windows in the Rust toolchain.
To ameliorate the risk, the security of all 73 dependencies would need to be at least an order of magnitude greater, just to catch up to shells with no dependencies.
The irony is that this lack of "dependency-safety" is far more easily exploitable than any previous lack of "memory-safety".
Of course, this can be easily fixed if developers stop multiplying third-party dependencies, so that importing a dependency involves O(1) risk, not O(?).
How is this an easy fix? Now developers need to develop and maintain O(N) instead of O(1) software projects.
I guess that if nushell had a zero dependency policy, it would have never happened.
And there is a significant risk to spreading your resources thin when reimplementing sensitive dependencies, like crypto or network / http libraries.
At some point, developers need to take responsibility for the security of their projects. If your project is larger, you will need to maintain more code, whether directly or indirectly. You can't simply import a dependency but never audit it. And if you're taking the trouble to audit a third-party dependency, at some point it becomes worth maintaining directly.
> sensitive dependencies, like crypto or network / http libraries.
I am not suggesting that developers reimplement TCP.
It's pain and sorrow all the way down and I find more bugs in my reinvented wheels than in the actu code I set out to write.
How about this:
0. Rebuild from latest stable dependencies, or latest updates to your LTSB, whenever you or your dependencies have a bug.
1. Prefer static linking or lib incorporation if feasible, or use a package manager that properly guarantees you the right versions.
2. Stick to popular repos with responsive maintainers, or
3. Fork the dependency yourself, audit the code and know how to debug it.
4. Watch upstream for security bugs.
Prefer dependencies with smaller flatter dependency trees.
Agreed, but that could be through careful selection instead of rolling your own. And it would be great if there was better tooling for that.
Imagine an ecosystem where every module author automatically gets not just a public repo and issue tracker, but also a way to deal with security issues, an indication if all contributors used 2FA for code contribution and package uploads. Then there could be a search for packages where you can say "only give me those with 2FA and a defined security disclosure process, including all transitive dependencies".
That wouldn't be perfect, but much, much better than any ecosystem I've seen so far.
The approach the Rust community is taking with a sort of decoupled-STL approach is interesting, but I do wish they found a way to collect the known-good-bits in a common namespace.
Or look at bash:
https://www.archlinux.org/packages/core/x86_64/bash/
I would say many shells that are written in C will have an extremely short list of dependencies in contrast to Rust.
As per your request:
$ ldd /usr/bin/bash
linux-vdso.so.1 (0x00007ffce8b8c000)
libreadline.so.8 => /usr/lib/libreadline.so.8 (0x00007fd76a0dd000)
libdl.so.2 => /usr/lib/libdl.so.2 (0x00007fd76a0d8000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fd769f15000)
libncursesw.so.6 => /usr/lib/libncursesw.so.6 (0x00007fd769ea6000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fd76a266000)
For completeness:
$ ldd /usr/bin/dash
linux-vdso.so.1 (0x00007ffdbadce000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f6f2cd28000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f6f2cf58000)
> What if we could take the ideas of a structured shell and make it more functional? What if it worked on Windows, Linux, and macOS? What if it had great error messages?
- Any shell with piping is already very functional, and PowerShell has script blocks (higher-order functions). Or does this mean "functional" in the literal sense? What functionality is missing? - PowerShell works on macOS, Linux and Windows - The examples of error messages further down look almost exactly like PowerShells error messages (underlining the command in the pipeline that failed)
It is not clear to me what exactly the authors sought out to do different
I often wonder if people just don't like it because all they see is PowerShell scripts, where people like to spell out commands so it's clearer what they do. Imo it's a nightmare that we write bash scripts with all these 2-3 letter commands and flags that no beginner could ever understand. (but if you want to write that way even in scripts, nothing is stopping you)
You'd have to alias every command and every option to make it enjoyable for interactive use.
And of course those aliases would need to be part of the standards powershell.
Which due to the naming conventions are still too verbose.
I tried using aliases instead, but there is a danger with this - some of the aliases are commands on Linux. For example, if I use the `ls` alias it will work as expected on Windows, but if you run the script on Linux, it will run the Linux `ls` command, rather than whatever the Powershell alias is for.
There is no excuse to make a new language that re-invents nearly every basic syntactic principle or borrows from the worst examples (bash, perl).
Although the naming isn't great either; the use of namespace prefixes rather than a flat namespace of long names might have been a better (and more logical) choice. It would have even allowed for better auto-complete.
To my perspective, making parsing for a variety of text-based formats a default action is an advantage, because it means tools don't have to use a particular runtime (e.g., .NET) to feed objects into shell pipelines without the shell user doing explicit conversion.
That doesn't mean there aren't ways to improve upon PowerShell.
As a light user of PowerShell I consider it a poor implementation of a great idea.
Commands are way too long, control structures have poor syntax etc.
To make this very concrete, take this example from nu:
ls | where size > 4kb
dir |where length -gt 4kb
ls | where length -gt 4kb
Nu doesn't seem to support redirection (or even have anything other than one input and one output stream, though metadata might serve the purposes otherwise served by additional streams; it's not how a program would return something Nu would treat as metadata, though.)
I guess that with a structured-data shell, error type of info could just be returned in-band under a different tag, in any case.
# most common in script, vs at shell
Get-ChildItem | Where-Object { $_.Length -gt 4kb }
gci | where length -gt 4kb
# longest most restrictive with namespaces,
# most stable over time,
# least risk of hitting the wrong command
# or wrong property as surroundings change
Microsoft.PowerShell.Management\Get-ChildItem | Microsoft.PowerShell.Core\Where-Object -Property Length -gt 4kb
# shortest(?)
gci|? le* -gt 4kb
# because you can
switch(gci){{$_.length-gt4kb}{$_}}
gci | & { process{ if($_.Length -gt 4kb) { $_ } } }
$larges, $smalls = (gci).where({$_.length -gt 4kb}, 'split')
[linq.enumerable]::where([string[]][io.directory]::EnumerateFiles($pwd), [func[string,bool]]{param($f) [io.fileinfo]::new($f).Length -in 0..4kb})
"Get-Process | Out-GridView -PassThru | Stop-Process"
Now, is it "Stop-Process" or "Set-Process -Status Stop"? "Out-GridView" or, say, "Show-GridView" or "Out-View -Grid"? The other examples are all technically legal, and other modules can and do have such inconsistencies.
With verb-noun, I can't meaningfully complete for the verb when I don't know it, I can complete for the noun if I know part of it. But I usually know the noun, so that's not so useful.
With noun-verb, I could get a meaningful verb completion and still be able to complete nouns - with some duplication of results, though I think there could be a way to have some semi-completion for the noun only. A semi-completion isn't useful in the verb-noun case, since the program can't guess the noun yet and different nouns may have different verb uses.
I really like the concept, but the slowness ruined the user experience for me.
My only comment is on the article. This line:
> Nu would not have been possible without Rust.
That's just patently false.
Perhaps you wouldn't have chosen to code it in another language. Or perhaps Rust had one or two benefits that assisted in the development of Nu. But you don't word it that way or go into any details as to what made Rust special/better than X, Y or Z language (not that you need to, but if you're going to make a wild statement, you should be prepared to back it up). The only 3 points made following that were about features that have widely existed in other languages well before Rust came along: async/await, async streams, and serde.
Rust is cool and is still exploring some really great ideas in making programs safer and more maintainable. But, it's not the greatest language ever (at least not yet), nor does it make the impossible possible. It can merely make some things easier at the cost of something else.
I'm glad you were able to use it effectively to create a tool you - and others - find useful.
I really like the idea and look of this project. To me it feels like PowerShell done better (and implemented by non-aliens).
I could put this together in Common Lisp. I know precisely what CL technologies I'd use to do it, and perhaps after finishing the work I'd say "this wouldn't have been possible without Lisp", which would be true in some respect, but just as dishonest as the original article.
That doesn't mean I don't respect the work done. It's an interesting project and most definitely something that deserves attention. The language used to develop it is the least interesting aspect of this.
Of course it is possible in any language you can think of, but not with their skills and time.
For example, the error reporting style in nushell appears to be a direct decendent of Rust's visual error message style.
If you factor in the real world, the number shrinks as you get to things like performance, cross-platform support, ease of packaging and distribution.
But still, Nu could plausibly have been written in Assembly, C, C++, D, Go, Nim, Zig, Haskell and so on.
You would have lost some of Rust's strengths and weaknesses and imported another set (regarding the language, tooling, performance, ease of debugging and so on).
I took the comment to mean that as far as they (all people intimately familiar with the language, tooling and ecosystem) are concerned, this would be a trade-off that would be negative enough to not begin or finish the project in the first place.
I don't know what plans and requirements the Nu authors had. But for my own game (a roguelike written in Rust) the requirements on the performance, distribution, stability, ease of development and dependency management meant that Rust was (at the time) pretty much the only viable choice for me.
Could someone have written it in C? Sure. Could I have written it in C? Likely, yes. Would I have ever finish the game in C? Almost certainly no. For various reasons none of the other languages would have done in those circumstances, not really.
Human language is an imperfect method for transferring information and sometimes one trades off brevity for nuance.
Programs like shells or servers are expected to perform well and usually cannot afford the GC tax (more memory usage and pauses). I wouldn't use a shell written in a GCed language. That would leave them with non-GC'ed languages. C or C++ simply don't have a usable equivalent of Cargo/crates.io which makes 3rd party reuse slower, and then they have much weaker execution safety guarantees.
Errexit (aka `set -e`) is a great idea badly executed. There are many things which can be improved (e.g. printing a stack trace by default), but at its core, there are two things which break it.
1. Inconsistent behavior: https://www.in-ulm.de/~mascheck/various/set-e/
2. You can't rely on it.
Inconsistent behavior should be no problem if a new cross-platform implementation is coming, but not being able to rely on it sucks. To give a quick demo:
#!/bin/bash
foo() {
set -e
false
echo "Sucks"
}
printf 'Normal: %s\n' "$(foo)"
printf 'Condition: %s\n' "$(foo || true)"
Normal:
Condition: Sucks
To be fair, I am not even sure if creating a new shell scripting language is something worthwhile in this day and age. There's some value in the convenience of instantly turning a sequence of commands you just manually typed into a script, but there's a conflict in that the shell has to provide you a way to hack a sequence of actions together with as few characters as possible, while a script has to be comprehensible, maintainable and extensible. And the inertia you would have to overcome to successfully introduce a new programming language in addition to a new shell makes it intractable.
https://www.oilshell.org/release/0.7.pre3/doc/osh-manual.htm...
Well actually there's one more thing I want to fix:
https://github.com/oilshell/oil/issues/179
That will fix your case, because foo is a function, and the disabling of errexit in || is indeed problematic there.
Please subscribe to the bug, or feel free to e-mail at andy@oilshell.org if you want to know when it's fixed. I have another person "on the line" (author of envdir I believe) who also wants this fixed.
It would be great to have you both try this out and verify that errexit in Oil has no more pitfalls :)
Now I’m really disappointed that it’s not called “nutshell.”
But this is still very cool.
elvish[0], uxy[1], ngs[2] and basically all shell projects that allow other languages(e.g. python: tako[3], racket: rash[4] janet: janetsh[5]) are all similar attempts; and there are numerous more alternative (non-structured) shells like fish[6], and a whole lot more.
As a daily user of fish and a person hyped by elvish (but not using it as a daily driver :-(), I hope some structured shells get at least some traction, but there are too much approaches.
Well, I didn’t start as a rant but it became one anyway.
[0]: https://elv.sh/
[1]: https://github.com/sustrik/uxy
[2]: https://github.com/ngs-lang/ngs
If I had to reinvent the shell by breaking backward compatibility, I would do a statistical study using all the shell scripts publicly available on GitHub, a survey about the critics of the current shells in the literature, and then I would try to optimize a metric such as : words typed to complete a task, reproductivity, time-to-market, communication, mathematical soundness, etc. The result of the study might be that the current shell is enough and the cost of switching to another is too big (I don't know, it's an hypothesis)
People shared many thoughts and opinions on this thread, so that's already a win by itself
Asking forgiveness is better than asking permission. Innovation rarely looks like a sure bet prior to it unfolding.
Yes, the real world problem of making it combining shell commands easier and more interactive.
A solution doesn't need to solve a whole new problem. It's enough that it solves an existing, already solved problem, better.
The same way the travel luggage with 4 wheels doesn't solve anything the travel luggage with 2 wheels or no wheels solves, just makes so much more easy and awesome.
https://ondigitalmarketing.com/learn/odm/foundations/5-custo...
What I feel could be improved is the presentation of the tabular data. Some ideas below:
- The screenshots show two horizontal lines consisting of dashes, one at the start of the table, one at the end. To me they seem redundant and could be replaced with spaces (whitespace). I did a quick check and PostgreSQL's psql tables also don't have lines like this.
- How about using Unicode lines instead of dashes and plus sign in the header? This way there would be fewer gaps there. I'm not sure though how this would affect compatibility.
Relative like "today" would be fine, if it includes the exact time today too.
I see you already have a where operator. What would fit this very well would be in-built sql-like syntax. Since tables are first class, you should have an in-built way of handling them.
A languiage similar to q/kdb - it has a great SQL dsl, but also works as a regular programming language. It's also very concise and expressive, you can write oneliners that do mind bogglingly complex things.
Really excited to try this, surprised that there is no pre-built binaries. Are these planned?
For instance, because of the influx of JSON the past few years I have learned to use jq very well. It does everything I need and I didn't need to upgrade my shell to use it. What if everything becomes protobuf down the road?
i feel like structured data in a shell will phase out.
The posix pipes allow streaming any data.
Nearly everything is built around a line-based philosophy. However, nearly any coreutil you care to name also likes to cram multiple bits of information on a single line.
That means you often have to do a bunch of ugly manipulation with awk, sed, cut, whatever to anything programmatic with the output of each command.
For instance, because of the influx of JSON the past few
years I have learned to use jq very well. It does everything I need
What if everything becomes protobuf down the road?
Nushell seems to build upon what we've learned in the last 30-40 years. Pipes and simple interchange formats work well and are very useful for a lot of use cases. However, experience also shows that very few programs output lines of text that are atomic in nature. Each line represents multiple things and we frequently need to parse them out. That seems like a very common use case that our shell can handle for us.
Personally, I'm not very accustomed to Python though, so I find the syntax kinda weird. I'd prefer something that feels more like a standard Un*x tool (like nushell).
The source is pretty interesting. Here's the PS command: https://github.com/nushell/nushell/blob/master/src/commands/...
And the sort-by: https://github.com/nushell/nushell/blob/master/src/commands/...
function both {
comm -1 -2 <(rg -i -l -t java -t go $1 | sort) <(rg -i -l -t java -t go $2 | sort)
}
As an aside, your one liner is having to process every file twice, even ones that have already been excluded by the other ripgrep... something like this should be more performant:
rg -il $2 $(rg -il -t java -t go $1)
rg -0 -il -t java -t go sharded | split-row "\0" | rg -il jobqueue $it
Instructing "rg" to use Nul as a separator and where "split-row" acts like xargs. (This only worked after I modified Nushell to recognize "\0" as the Nul separator. It's surprising that this case hasn't come up many times previously, so maybe i just missed the proper way to do it).
Edit: Of course, not many filenames contain the newline character embedded in them, so usually this would be fine in an unmodified Nushell:
rg -il -t java -t go sharded | lines | rg -il jobqueue $it
Seems clear after some use that Nushell has a pretty long way to go to be a practical option.
> There is no spoon^H^H^Hstructured data. There is only text.
^ls -la | lines | skip 1 | split-column " " perms files group user size month day time name
e.g.
myls = ^ls -la | lines | skip 1 | split-column " " perms files group user size month day time name
myls | where size > 1000
Sure, and they'll get "ls" (the standard one, from POSIX), which should be the least surprise. From then on, they'll know to call nu.ls explicitly when they need it. Sounds like a better option.
Pipelines Support Vectorized, Point-Free, and Imperative Style
http://www.oilshell.org/blog/2017/01/15.html
You just do
myls() { ls -la | lines ... ; }
my ls | awk '$1 > 1000'
The awk part isn't ideal, but I have some plans to fix that:
What issues could be faced with this approach?
Having a toggle creates the problem that you have to write two implementations for your tools, or you end up having to memorize which commands can run in which modes. It's easier to have a consistent pattern and a way to up-convert existing CLI programs.
[1] https://www.gnu.org/software/emacs/manual/html_mono/eshell.h...
However an important point for practical usage is the capability to run existing scripts, for example as oilshell [1] does. I skimmed through the README and doc but couldn't find the current status nor future plans.
Obviously it doesn't necessarily mean the project has no merits, but language choice is one of the less relevant details and it signals that the author wanted to make a toy in a language new to them. Good for them, but most of the time the rest of the world doesn't care.
If the project has merits, and this one might, it should talk about them in the intro.
Much of the time, HN readers however will. There's the odd software developer hanging about here, some of whom are interested in programming languages, or so I believe.
I don't think I want the fad to go away, because it can even help me know what to steer clear of. e.g. Introducing a new shell written in JS!).
I have to say that I don't care for the snark about "having to remember" arguments to various pipeline-munging commands. I get that this is a big challenge for new folks working with typical unix shells, but I'm not sure I see how this actually solves it. In order to avoid all those flags, a system like this requires more verbs overall, and those verbs have their own argument structure, eg `split-column " | " firstname lastname job`, so now instead of being able to label the arguments and order them how I want, I now have to remember the order they come in, and what they mean, and that `split-column " "` apparently eats extra whitespace (does `split-column " | "`?). Yes, the standard unix tools have these issues, as well. That's my point.
So yes, I love this idea, please keep pursuing it, and I look forward to trying it out. But focus on the benefits of working with structured data rather than pushing lines that sound good but aren't actually true.
I'm finding it hard to get over them using pipes and dashes for table borders. Is this the 70's? It's genuinely hard to read these tables. See: https://www.interaction-design.org/literature/topics/gestalt...
e.g. `ls | nu where size > 10` or `nu open Cargo.toml`
Can I make nushell cmdlets (or equivalent) in something other than rust? pwsh limits me to .net languages, but I wanna use node.
https://www.ccs.neu.edu/home/tov/code/shcaml/doc/
Edit: features look similar in fact, since you can name columns and refer to them by name. Neat!
I'm interested to see a solution to error handling (within multiple pipes!) and subprocess management as well but couldn't find much in the docs.
https://github.com/python-mario/mario
$ mario read-csv-dicts map 'x["name"]' < hackers.csv
Alice
Bob
CarolIs anyone interested in a shell with a GUI? As in, data still passing as text, but having a shell which can render textual data into widgets when displayed in the shell?
My favourite feature is that you can select objects in the GUI to pass on to the next pipeline step
Really interested to see where this goes..
There's an elegant simplicity of treating all piped output as a string rather than objects...
I also thought quite a couple of times about implementing a data centric, typed shell language. My primary goal was to have a shell where one could discover every feature by oneself through the magic of types.
Would you pay for this?
My prior is that no, people would not pay for a new shell - they'd prefer sticking to bash because A. it's "standard" B. it's free.
I have to wonder how developing new tolls like nushell can be sustained over the long run.
Doesn't “open --raw” address that?
https://book.nushell.sh/en/loading_data#opening-in-raw-mode
Also, I suspect most simple scripts that want to work on JSON Will want to parse it, so the default makes them more easily able to work with it, not less.
gc foo.json | cf-json
Get-Content foo.json | ConvertFrom-Json
Unfortunately, I'm having trouble building it on Mac OS X 10.14.5 and with the latest rust update installed.
cargo:warning=libgit2/src/streams/stransport.c:12:10: fatal
error: 'CoreFoundation/CoreFoundation.h' file not found
cargo:warning=#include <CoreFoundation/CoreFoundation.h>
I desire a fancy new shell but don’t want it to become a crutch, limiting my usefulness and familiarity when I’m on a system lacking it.
Also, using structural regexps for efficient interactive text editing: https://github.com/martanne/vis
In the real world folder hierarchy is probably inevitable so yes, it has to be stored in a way (in a separate table perhaps so we can have many-to-many relationships sort of what hardlinks do).
I think you'd like prolog https://swish.swi-prolog.org/example/examples.swinb
One benefit of folder hierarchies (and I don’t think there are many) is that it’s easy to delete/move hundreds of thousands of tiny bits of associated data just by deleting/moving the parent.
The second nice thing was the new taskbar (with pinnable app buttons and without text) in Windows 7-10. I've even configured the taskbar in KDE (which I use at home) to the same mode.
The third cool thing is cross-app text input autocompletion and correction made available with physical keyboards in Windows 10. Sadly I don't know how to accept a completion suggestion without using mouse, yet the feature already helps me a huge lot (as I write a language I'm not very good at).
If not these 3 things I'd be glad to keep using Windows 95, if only there were security&stability updates and current software and hardware support - newer versions of Windows didn't add any other features I would find useful. Everything else (besides internals which have obviously improved a lot) is either bloat or a minor goody.
Windows 10 has seemingly added universal file tagging support (previous versions could only add tags to files of some formats which allow injecting them in file body) but as long as Windows, KDE and MacOS don't store the tags the same way (so I could tag files on an external drive using Windows and keep using the tags when I attach the drive to a Mac or a Linux machine) this is of little use for me.
They are already tables after all :-)
What Is a Data Frame? (In Python, R, and SQL)
http://www.oilshell.org/blog/2018/11/30.html
I think starting with either SQL or data frames would be a good choice. They are similar except data frames also focus more on mathematical modeling, and data frames usually don't have indices. Otherwise they both have selection/projection, sorting, various kinds of joining, etc. Data frames also have vectorized arithmetic.
I am trying to reuse the designs that have proven to work over time (R, SQL, bash) rather than inventing new variants with potentially needless differences.
Fselect has few really cool features, esp. when you got to deal with complex queries. Although not unix way at all.
We use a similar approach in our data prep tool [1] for non-technical users. Lists of files, lists of emails, contents of text files, spreadsheets or database tables - piping it all together allows mixing ETL and process automation in a similar fashion.
Disclaimer: I'm the founder.
https://github.com/nixpulvis/oursh
`nu` seems very cool (and I love the name ;), but isn't building atm.
This seems to have a lot of the power of Powershell, but with a terse, familiar syntax - I really like the look of this!
Being cross-platfotm is a huge seller for me too - I mainly work on Windows, but also on Linux and MacOS.
Can you imagine the cute squirrel logo? Mouth full of peanuts? Nushell sounds like an new wave band.
Previous HN entry about it: https://news.ycombinator.com/item?id=15600928
> or why do you get
> command: arg list too long
> [...]
> It's only the exec() system call and its direct variants, which will yield this error. They return the corresponding error condition E2BIG (<sys/errno.h>). The shell is not to blame, it just delivers this error to you.
https://www.in-ulm.de/~mascheck/various/argmax/
Additionally, see also https://unix.stackexchange.com/a/120842 which speaks a bit about Linux specifics, and how Linux behaves a bit differently from a lot of the other operating systems in the Unix family with regards to limits on the length of arguments.
I've been using fish shell for years and glad I started down that route. It's been incredibly useful.
I really want to give this a shot too. I feel like too often I'll open up Python, read a file and then manipulate it when I need to something more complex than simple grep/sed/cut/wc etc... This feels like it really fills in that gap. It starts with something really simple: get your unstructured output in a structure, and then build from it. I could see people writing more custom extensions and objects later on if this gets popular.
It's spectacular to have a shell with consistent verbiage and parameters.
The problem is that it treats any output to stderr as an error and ignores actual posix errors (non-zero exit codes.) What's worse is in some contexts the error it throws on stderr is impossible to catch.
I really love Powershell in a lot of ways but this makes a lot of really basic scripts in bash very cumbersome to write in Powershell.
[Remoting Bug] https://github.com/PowerShell/PowerShell/issues/3996
[RFC for improved error handling] https://github.com/PowerShell/PowerShell-RFC/pull/88
Ultimately I just want Powershell to handle well-behaved posix apps in a sane way, but honestly the reaction of some of the people on the thread seems like there's a huge cultural gap between Microsoft and the Posix community. (Which, Microsoft claims to be a part of but clearly is not.)
Anything you can script in PowerShell can have unit tests and code coverage and it integrates into CI systems.
I don't really like using PowerShell, but built-in unit testing is something I wish other shells offered. It is a huge plus.
Do you want to live your entire life and everything you do in sql? I don’t.
I think I will pass.