The Dawn of a new Command Line Interface (2017)
arcan-fe.com
arcan-fe.com
Besides stdio and stdout, one of the best-accepted conventions on modern operating systems running a graphical shell seems to be the clipboard.
Programs can offer different mime types trough a clipboard, so why not do so trough a pipe or file descriptor? That would bring us a bit closer to powershell, while remaining backwards-compatible.
cat pipe.dot <- text/plain
| dot <- text/plain, text/dot, image/svg+xml, application/pdf, application/postscript, etc
| xclip
xeyes | ssh somehost
record-audio.sh | mail someone@somewhere
nmtui | guitoolkit
rsync | guitoolkit
ssh somehost rsync | guitoolkit
Imagine:
> # See alternate data streams
> Get-Item diagram.dot -Stream *
Stream Length
------ ------
:$DATA 123
text/plain 4444
image/svg+xml 65432
image/png 456134
> # Send the whole bulk
> Get-Item diagram.dot -Stream * | ...
> #... or pick just one:
> Get-Content diagram.dot -Stream "image/svg+xml" | Send-Email ....
I agree that both ends need to be aware of these extra formats to make use of them, but just like with clipboards, programs could always fall back to pretending only text/plain exists :)
How can a shell do anything serious with asserted file types that can be missing, wrong, malformed, unknown or mendacious? It's hard to go beyond offering a best-effort "open with the associated program" application as an alternative to vi, cat etc.
How would that be any worse than the status quo, though?
Programs already output, depending on their invocation and the utility at hand, binary data in arbitrary formats, shell escape sequences, text, ASCII art, etc.
The only issue I can think of is for portability of the scripts, if a program gets updated to support a new type as input or output, and that causes issues compared to the testing environment. But that's arguably already an issue too, and it would be rather simple to annotate expected types.
https://en.wikipedia.org/wiki/NTFS#Alternate_data_stream_(AD...
/..namedfork/rsrc
ls ./groovyFontFile/..namedfork/rsrc
I've tried it with ancient font suitcases -- the ones that pre-date dfont (data fork only) fonts.
It's somewhat ironic that all desktop file systems these days support myriad data streams on a file. I've implemented some on NTFS, played with extended attributes across tar archives between Linux and Mac and Windows. It's all there, but no longer used.
The problem isn't so much finding a way of passing metadata but how do you retain compatibility with existing GNU / UNIX tools? It would be easy to create an entirely new shell (like Powershell) that throws out backwards compatibility but that wouldn't be very practical on Linux/UNIX when so much of it is powered by old POSIX standards.
I addressed this by having a wrapper around pipes. Anything that connects out to a classic executable is a traditional POSIX byte stream with no type data sent. Anything sent to a command that is written to support my shell can make use of said meta data.
As for handling graphical components, that's a little easier. Many command line programs already alter their behaviour depending on whether STDOUT is a TTY or not. I take things a little further and have a media handler function, called `open`, that inlines stuff in a shell that can be (it will render images in the terminal) but if STDOUT is a TTY then it will pipe the image data instead.
> The problem isn't so much finding a way of passing metadata but how do you retain compatibility with existing GNU / UNIX tools
Honestly? I'd have suggested a new syscall that provides an interface similar to isatty(). It seems a matter of adding a few constants to the ioctl syscall, prototype it and prove it is useful, then upstream it. Better make a crude prototype, see if that's any useful, and ask upstream what they think of it.
If the new ioctl isn't available, nothing changes, programs continue exchanging whatever they expect. If the new interface is available, most programs won't know about it, and would continue their business as usual. However, programs that know about it could get more metadata trough it, and communicate in a smarter way. Looks like a standard way of introducing new functionality, can be deprecated if needed, and keeps backwards-compatibility, at the expense of a tiny bit of complexity (dwarfed by having to support multiple formats, which some tools already do).
The way you do it seems quite fine too, not sure which would be better.
Staying with traditional file descriptors also has the issues that you are limited to unidirectional communication, and only one file descriptor, so maybe you need a protocol like html, as a sibling joked.
> As for handling graphical components, that's a little easier. Many command line programs already alter their behaviour depending on whether STDOUT is a TTY or not.
Well, that's kind of my point: you could output more than curses interface if you wanted, something like an xml ui file, or a list of commands on a side channel. But I make no judgement on this: it was just to illustrate what kind use-cases this would enable.
So with those points in mind I think this is one of those cases where worse is better. But kludging things a little it means I can have greater ownership of development which means I can actually deliver something that people can use right now. If it gains popularity then maybe I'd have more clout to push through new syscalls, but I certainly wouldn't have that with "only" 500 Github stars :)
> Well, that's kind of my point: you could output more than curses interface if you wanted, something like an xml ui file, or a list of commands on a side channel. But I make no judgement on this: it was just to illustrate what kind use-cases this would enable.
You can do that with my shell right now. The issue is you'd have to write an `open` handler (because how would the shell or terminal know what to do with pseudo-random data if you don't have some kind of handler?) but PNGs, JPEGs, etc will be rendered with sixel/or whatever proprietary support the terminal emulator has for inlining images if `open` is used and STDOUT is a TTY. So it's not just curses interfaces though the principle is similar.
In one experimental module I even went as far as having web pages render in the terminal if STDOUT was a TTY. So instead of making a HTTP(S) request and displaying the bytes, it would render the webpage, images and all. Only falling back to a standard `curl` operation if the command was piped. This never made it to production code though, it was very Linux specific code and required a heap of 3rd party software to be installed just to work. But it was a great example of just how much you can already do with some terminals.
This blog from years ago describes something similar: http://acko.net/blog/on-termkit/ i.e. expanding the process model
I thought a bit about this, but that becomes a bit complicated without additional metadata on a file descriptor (is it really that additional interface, or just a random FD we inherited?). It might be possible to achieve in a robust way with mount namespaces and named pipes, but that's another can of worms...
That said, it would likely work 90% of the time that way.
Integrating some new shell projects like murex and nushell with Extraterm is on my TODO list.
That said, why not use env vars which tell the child process what to expect on which file descriptor numbers?
Depending how crazy you want to get, you could also use unix sockets to pass around FDs instead of inheritance. In that scenario, a process would be passed a unix socket either by a socket pair file descriptor or the shell would have a named socket which it tells child processes about using an env var. You can then use the sendfd functionality of unix sockets to pass around arbitrary file descriptors with arbitrary metadata.
You could also use the unix socket to instead have the child process optionally ask the shell what the descriptor's metadata is. That way you'd keep backwards compatibility with unix tools, but the child process could sendfd the file descriptor back to its parent to ask for data about it. The parent could look it up by calling fstat on it and comparing the fstat data to the other file descriptors it sent to the child.
But regardless, when you can't add a new syscall and need to do file descriptor stuff on behalf of another process, unix sockets + sendfd can really start to look like a userspace syscall interface which you can do a lot with.
That would be quite a change from the traditional pipe model.
That wouldn't change much from the pipe model. At least on wayland, wl-copy forks to the background. If you wanted the producer to finish producing data and exit, surely you could make a flag for it in the receiving program, or pipe it trough one that does.
Also, in your example ending in | xclip, xclip cannot know what format data it has to ask from dot until it gets pasted somewhere. Alternatively, it would either have to guess what format(s) the user would like to paste later or it would have to cover all bases and ask dot for output in every format it supports.
first | second | third
Plus the pipeline then has ongoing two-way communication up and down it.
Obviously the following can't be taken at face value:
> lightweight headless Chrome Electron [...] with only a small overhead of cookie policies, cross pipe scripting, modern auth tokens, 2-factor authentication between client and server, SAML, and a tiny surface area increase for security concerns but only a few dozen MB or so.
However, there's probably much to be learned from the history of the web when it comes to security and compatibility between different implementations and successive evolutions of a protocol. http certainly has merits, given its long history. It isn't flawless either, of course.
Even features such as compression could make sense at first glance, though they would be a better fit for a dedicated tool like piping to zstd, since you control both ends of a pipe. Range requests, multiplexing, server-sent events, etc, could have some use. And it isn't like making a http server or client is so hard with a small library, given you already pull in libc or worse.
http probably wouldn't be suitable as-is, certainly not over TCP (though the fopen/fclose dance is quite close to it), but making it a fully fledged bidirectional protocol could be interesting, especially if you can somehow build on top of the system IPC on a microkernel. It could be a way to forward hardware access and such.
Can you tell me more? So I know what to expect when I hit such issue myself.
> The work around is to write a wrapper in a compiled language and use PowerShell to invoke the wrapper...
Wonder if you need to really break out of PowerShell for this.
Recently when doing some scripting at work, I ended up writing .NET code straight in the script. It looks like this:
$someCSharp = @"
using System;
//...
public class Foo : Bar {
public override object Baz(Quux arg) {
//...
}
}
"@
Add-Type $someCSharp -ReferencedAssemblies C:\Path\To\Some\dependency-I-need.dll
I suppose using a proper .NET development environment for this would be more kosher; I arrived at the solution described above because my needs were simple, like binding .NET regex engine to SQLite (to get REGEXP operator in queries), or convincing Invoke-RestMethod to stop checking certificate validity.
Still, I love the capability, the fact that I can just drop into C# in the middle of the script. It's one of those things that make me think of Lisp machines and Common Lisp the experience of being able to deeply interact with your environment, and the joy of writing "scripts" that are automatically AOT-compiled (like with SBCL implementation of Common Lisp).
OTOH, PS 7.0 added ForEach -Parallel
The fabric under attack was the bastard display server protocol itself and not the parts that tie in with the strong points of Powershell, i.e. shell UI design or data exchange format specifics.
This story was continued in a post from yesterday that fizzled, https://arcan-fe.com/2021/04/12/introducing-pipeworld/#pipel...
Where Arcan cannily (IMHO) differs from Powershell is it acknowledges the transition from existing toolchains will be organic instead of demanding wholesale operational changes to a different processing model. So it splits the data and metadata processing pipeline, exposes it at key points to the developers using Arcan, and lets an emulation layer cover legacy use cases, while enabling Arcan developers to create much richer data representations underneath the covers that can be tapped if desired by users via Lua manipulating the many different Arcan components (I think). This opens up hybrid GUI/TUI/other-interface best-of-breed/mixed-mode options that are unreasonably challenging to implement (especially with sync logic) with current UI stacks.
I like this approach more than Powershell's, but it remains to be seen how Arcan handles the guidance of the data and metadata ontology and resultant taxonomy; it is a much deeper problem space than just defining a default namespace, a way to modularly plug into it with extensions, then standing back and letting developers have at it. The way Powershell handled it, introspection and discoverability are out of band instead of a batteries included experience, and module management is still split out of band between Nuget, PrivateGallery, custom solutions, hand-curated Git+Jenkins lash-ups, etc.
For my small-scale, personal projects like for my family's intranet, the lack of an opinionated default for at-scale management of these issues is not a problem. But at my clients with dozens, hundreds and even thousands of developers, it presents software engineering management challenges. Solvable, just a developer experience hygiene factor I'd rather not deal with because solving it adds zero business value, it's just the cost of doing business.
I could see a whole library of such wrappers being built for common executables (and new popular OSS CLI tools could provide their own wrapper along with the tool).
Oh! Wait! This is what tools like cut, jq, grep, awk, and sed to.
And they won't tie you to a certain object model.
Yes! Except, you have to use them to write one-shot parsers that are specific to both what you're piping from and what you're piping to! For n commands you may want to arbitrarily mix, that's n² parsers to write.
> And they won't tie you to a certain object model.
From the point of view of a system user, tying to a certain object model is a feature. It facilitates commands interacting well with no parsers in between. But even if you don't have a common object model, just wrapping the commands to produce structured output can already be helpful, because the parsers you'll be writing to glue commands will operate on structured, higher-level concepts. It's easier to map between two different representations of what a "file" or "process" is, than it is to manually dig them out of an unstructured text stream.
To use a stupid example, assuming hypothetical "kill-and-announce" that echoes a process name and sends a SIGKILL to its PID:
ps aux | awk '{ ...extract columns... }' | sed 'rename columns' | xargs kill-and-announce ...
wrapped-ps | wrapped-kill-and-announce
wrapped-ps | select Target={id:$_.PID, name:$_.name} | wrapped-kill-and-announcePersonally, I’m discovering more and more that CL’s REPL in SLIME is my preferred shell in part because the CL ecosystem has a rich array of operations on a small number of data-types (without forcing everything into binary streams), but also the ability to define ad-hoc datatypes as needed and a powerful system for manipulating them and integrating the custom data types into existing protocols.
Also, the claim that you need n^2 encoder/decoders isn’t actually true: conventions around data formats can reduce it to O(n) in the number of conventions.
Myself I know I'm stuck here. I've been thinking in circles for a while. At the moment, I'm leaning towards rich type system and structured data, but I'm not sure if it can be made fully convenient for the shell.
Do you know if anyone attempted to summarize this debate and distill some conclusions, or reframe it in a way that would allow to make progress?
> I’m discovering more and more that CL’s REPL in SLIME is my preferred shell
In terms of command line or scripting? SLIME is awesome, particularly with its interactive Inspector :). And I wrote my fair share of "scripts" as CL functions. But I can't seem to get used to using the REPL as a shell; I feel something is missing. Most likely convenience functions (that I didn't bother to write for myself yet).
That whole paragraph you wrote applies to another Lisp system, though: Emacs. In recent years, I've been leaning on it more and more for day-to-day operations. For instance, I rarely use shell for file operations - I mostly stick to dired. I end up scripting common operations straight in Emacs Lisp, and bind them to ad-hoc UIs (using the Hydra package). And of course utilize well-known packages like Magit (for Git). This recent experience makes me think that we could pull more common shell interactions and turn them from 1D to 2D interface. And I keep dreaming about merging it with PowerShell, in a way that allows Emacs to ingest and manipulate .NET objects nicely, while at the same time exposing its internals to PowerShell.
> Also, the claim that you need n^2 encoder/decoders isn’t actually true: conventions around data formats can reduce it to O(n) in the number of conventions.
I think this assumes conversions are lossless, which isn't the case in UNIX piping. When you invoke "tools like cut, jq, grep, awk, and sed", it's almost always to filter out most of the text in the pipeline.
The experience could be replicated in UNIX shells, but noone sems to care to actually go down that path.
Honestly? Powershell is pretty cool, though it feels a little strange to admit that. It has it's own very powerful syntax and virtually all of the common Bash commands are now aliased by default, so it's pretty smooth transition for the basic stuff.
What a low value and needlessly dismissive comment. And based on HN's track record it probably means Arcan actually has a shot.
I think the point here is that DOS explicitly rejects the composition that pipes enable, in exchange for the ability to just hand "the whole system" over to a programmer.
But the main point, that DOS was basically a thin program loader that "got out of the way" is sound. Think of it like a UEFI shell environment, with 640k+ limits.
There's been a few attempts at this, but I settled on the fish shell as it has a good balance and syntax highlighting for interactivity.
I still write simple scripts in sh and complex ones in Python however.
It's from The Art of Unix Programming (2003) by Eric S. Raymond. It's actually in the first chapter. And the supposedly original quote is:
> Those who do not understand Unix are condemned to reinvent it, poorly.
> -- Henry Spencer Usenet signature, November 1987
http://www.catb.org/~esr/writings/taoup/html/philosophychapt...
Henry Spencer being: https://en.wikipedia.org/wiki/Henry_Spencer . I imagine that's why the quote is also in that regexp book.
It's a super arrogant quote used, I'd say, 95% of the time by people who don't deeply know other OSes (besides shallow memes and jokes about them).
The fact that the overwhelming majority of great applications, services, any kind of software we create is not written in Lisp should at some point in the first century after the creation of Lisp, lead Lisp afficionados to some kind of enlightenment, one would hope.
Unlikely, since it's already been 60 years, so they only have 40 years left.
No, CL's popularity hasn't changed over the past 30 years.
Looks like Burke lived a century+ earlier.
It's easy to make a laundry list of the ways the traditional unix shell falls short. The problem is creating an alternative that's superior enough to warrant learning something new and consensual enough that it can get mainstream appeal.
The article acknowledges that:
>There is arguably a rift here between those that prefer the ‘shove it in a browser’ or flashy UIs that animate and morph as you interact, and those that prefer staring into a text editor. It seems to me that the former category gets all the fancy new toys, while the latter mutters on about insurmountable levels of legacy. What I personally want is many more “one- purpose” TUIs and for them to be much easier to develop. They need to be simpler, more consistent, obvious to use, and more configurable. That’s nice and dreamy, but how are “we” supposed to get there?
But then I argue that they jumps the shark a bit. As far as I can tell the Arcan project seems more preoccupied with reaching feature-parity with Xorg than actually improving the CLI. And I'm sure for some people that's great, but for me personally that makes it a funny novelty thing that I don't see myself using. It reminds me when I first saw compiz ~15 years ago, where you had virtual desktops on a rotating 3D cube with all sorts of weird visual effects[1]. It looked cool, but I never actually used it because it wasn't practical and I had no use for it.
On the other hand I'd love for a better CLI implementation, better pipes, easier debugging, a more expressive scripting language, better task control, better feedback.
But then we went full circle to the core of the issue: everybody agrees that the unx CLI sort of sucks, but we don't really agree on what exactly needs fixing.
As a graphics effect it's about as basic as it can get -- even the venerable GLX Gears is more complex. And I found it to actually increase usability because it visually and very intuitively showed you the movement, so it made organization a bit neater. It made it easier to keep a mental map of what was where, because it could be arranged spatially.
If anything, it didn't go far enough. I wanted more faces, and the possibility of working in a "zoomed out" state, for when I need to look at a whole bunch of stuff at once.
The real novelty stuff to me were the Enlightenment effects like the water effect on the bottom of the screen -- now that chewed up CPU and did nothing of actual use.
You could make the 'cube' have an arbitrary number of faces, but I don't think there was the zoom mode you talk about.
You could add more faces to the cube so that it became a slice of a pentagonal prism, hexagonal prism, heptagonal, octagonal etc. Or take one away so you were working on the sides of a triangular prism.
But you're right, there was no vertical freedom of movement. You couldn't use a Rhombicuboctahedron, cool as it might be :) (https://en.wikipedia.org/wiki/Rhombicuboctahedron)
It would be interesting if there was an Arcan-philosophy UI framework in CL for an updated Climacs on top of a bare metal Lisp environment, for a turtles-all-the-way-down CL experience. It would open up interesting interactions with Emacs that are an unreasonably big pain to implement today, like buffers that manipulate view ports into observability data under a pluggable UI that renders graphically a graph data-oriented view of the discovered structure of the data under an investigation taking notes into a Jupiter-like notebook with rich multimedia data, for instance.
The pieces and principles force me to think about many of the "skills" I have acquired in order to work on various shells and desktops as burdens.
I can't bring myself to change to (or develop) pipeworld/durden-esque environments (yet?), I am drawn to them because I have a strong, natural alignment to the principles the author espouses and develops against.
Keep up the good work.
Here is a related issue. There are many cousin issues.
Intuitively I feel that we can do better on the presentation side. It is also tempting to refactor pipelines to use structured data - at first sight this would help avoiding all this plaintext parsing glue which hides otherwise elegant logic in your pipelines.
However, digging deeper into possible solution frustration creeps in :( In my case it started when I tried to collect real world use cases for "rich terminal":
cat image.png # that is pretty straitforward
cat *.png | gallery -3 # displays picture gallery in 3 columns
cat some_data | plot .... # hm... not that I use this frequently
.....
psql -c "select * from customer;" # hm.. I do need some kind of grid control here.
Heck, do I really need to bring such a complex thing here? Ok, I'm a bit stuck, lets postpone this and think about the implementation.
The obvious choice to make some shortcut such as HTML. However, here comes JS and I'm totally unsure that this behemot is what I wanted. Improving VT100? Most likely this will end up with a brand new 15th standard. No luck, lets postpone.
Structured data. Lets scratch another itch - model some basic commands using Clojure to build commands and represent structured data. ls, find, cat, grep. After a good start with producing file tree, flattening and filtering it with core Clojure function it becomes clear that I need some kind of DSL for "find" so that it would be as concise and easy type as with "find".
Блин!!!! Lets wait for the next such post on HN to start over.
It's not as fluid for me as a POSIX environment, 35 years of shell muscle memory.
PowerShell is a bumpy road: some commands are super powerful, and some bits have strange limitations.
But it's certainly encouraging that people are getting stuff done.
By and large, I like knowing that I am in control of what I send a program on the terminal. It is actually maddening when I go to paste into a word processor and it doesn't just bring over the text. Or when I go to script a program, but it doesn't act sanely when driven by a script instead of by a user. Where sanely is, exactly the same.
There are times when I have taken advantage of eshell to inline images and such. That said, I don't use many tui style programs, other than emacs. And I greatly appreciate that the majority of emacs buffers are textual buffers.
For something completely different: https://hisham.hm/userland/
Terminals are decoupled from the actual application for a good reason...
if we want to get something better than current terminals, a new protocol and client are needed.
take mosh for example. it uses a different protocol than ssh too, which adds some interesting features. and it replaces ssh (except for relying on ssh to make the initial authentication)
there is no reason why another alternative to ssh could not be created. if it's good, it will catch on. just like mosh which is slowly growing, and also like ssh was in the very beginning when it slowly grew to replace telnet.
While, you know, ssh carries your bits and the terminal displays said bits, just like it displayed the telnet bits before. Decoupled is good.
also i am not disputing that a new system could not be decoupled. i am just concerned that it should not be required to work with ssh if that becomes a limiting factor.
architecture astronautitis is certainly bad, but maybe it's possible to create an implementation with a limited subset of features that are actually practical.
> Recall that the prime target now is local text-oriented, command line interfaces – not changing or tampering with the awk | sed | grep | … flow, that’s an entirely different beast.
So I'd call this a "TUI" (text user interface), and the proposed schema replaces X11 or Wayland, not xterm.