Hacker News Reader

Comment by an1sotropy | Hacker News Reader

One thing not discussed are the libraries used for command-line parsing (parsing argv), and how that might get complicated by shells trying to make the command-line into something effectively more than an array of strings.

Having written a non-trivial command-line parser in C, and having used a bunch of them in other languages, it seems to me that this task would benefit from some more standardization and maturation. What is the JSON of the command-line? How can we do to increase the level of interoperability between how information is encoded on different tools' command-lines? e.g. think of ImageMagick "convert" versus "find" versus "ffmpeg": totally different universes, but all of them in their own way turn command-line arguments into mini-DSLs.

View on HN

Given the prevalence and longevity of GNU style short and long options, pretty much anything that doesn’t follow that is “out of compliance”.

However, you also called out some very specific commands that are that way for a reason. For example the order of options for ffmpeg matters very much, as that’s used to construct the processing pipeline. It does make sense for certain things to be custom, but that should only be done when there’s a good reason.

It doesn't scale especially well to UIs with tens of subcommands, but I'm a fan of Docopt as a reasonable way to write basic CLI interfaces in many languages with a minimum of fuss:

http://docopt.org/

I have a hard time imagining how we get out of the gravity well of CLI programs handling their own parsing.

A tool I write has a use-case for understanding the syntax of at least ~common CLI tools well enough to pick out args that will be other executables (sudo cat, find blah -exec...), so I have been idly pondering whether there's a humane, declarative, descriptive grammar that can express nearly all CLI interfaces.

It's probably not worth the work for my case, but it might get to be more tractable if it was also an input for better completion, help, linting, etc. tools.

Ideally something that drives enough all-around value that projects would start up streaming the grammars (and maybe adopting an associated parser?)

You might find http://docopt.org/ to be of interest. (It's available in many languages https://github.com/docopt).

Parsers designed for implementing CLI programs are generally too opinionated to handle ~strange commands. (In my terms I'd say it's a prescriptive parser as opposed to something that attempts to be flexible enough to describe nearly all existing CLIs).

Wouldn't it be easier to have a convenient library/parser for almost all of the use cases instead of an immensely complex catch-all solution? Having custom logic when required should almost always be less complex when such a ~strange command is to be implemented.

As I said, my usecase doesn't involve implementing the commands--it involves reliably identifying executables in the arguments to many different commands.

I can't go rewrite awk, find, and sed with an opinionated cli module. I have to deal with the current reality.

(you're roughly describing what I already do, and it scales poorly)

You might find Fig completion specs useful for this: https://github.com/withfig/autocomplete

Completions have in general been of interest, though the shell-specific completions I've looked at so far were all too dynamic.

I'd forgotten all about Fig since I saw your launch post here last year, so thanks for reminder. (I don't think I had quite started to work on parsing specific external commands, yet, so it wouldn't have clicked at the time. Was still focused on just identifying the likely presence of exec in the executables.)

Are you familiar with the parse code? Are you handling painful stuff like combined short flags with a trailing option? (If I ferreted out some of the more painful cases I've had to wrangle, I am curious if you'd have a gut sense of whether your approach handles it. Would you mind if I reach out? I am working on this for https://github.com/abathur/resholve)

I've always wondered about expanding stdin, stdout, stderr. Say, stdjson that doesn't get visually displayed, but can be piped (and would only be generated if it is needed on the pipe stream).

ls | cat </dev/stdjson | string_proc_the_json_for_some_reason

With the direct ability to process in line:

ls -a | json.files[0].last_modified

I'd probably want multiple output formats (including s-expressions).

You can always just use JQ.

Additionally, as JSON is text, you can use awk/sed/grep and so on.

The intent is that standard tools would output alternative formats.

Right; every file is just bytes but we get a lot of mileage out of libraries like libpng that parse those bytes into usefully structured info. And I was pondering what more could evolve to parse info from the command line.

I think stdjson is insane yet awesome to ponder.

I agree that a standardized structured format would be awesome, but I'm not convinced that JSON is it. And there should _not_ be more than one.

One thing that I do like about JSON is that it is ubiquitous - and that makes up for a lot of its other faults. But I would like to see proposed use cases that JSON would not support beforehand, to clearly define where the limitations are, and what limitations the community is willing to accept.

The tool should instead provide an `--output=<format>` flag, where <format> is one of text (default), json, xml, etc.

(fully daydreaming here) For C, would it be useful to have main() look like:

   int main(int argc, char *argv[], char *envp[], JSON *json)

for some JSON data type that is part of C, kind of analogous to a FILE stream? I'm not sure how the the json info would get into that fourth argument (has to be independent of argv), but it would keep std{in,out,err} as is.

You would need to modify the execve system call (or equivalent non-Linux OS's) to take in the 4th json arg for the shell to pass when it exec's the new process. And then of course modify the OS kernel to parse/deal with it. But in reality, since you can't break the ABI like that for all existing software, it would end up being execve_2 or something and you'd have to both convince everyone else it's worth using, and deal with the inevitable incompatibilities when not everyone does. Not impossible perhaps, but certainly an uphill battle.

It sounds like you either want powershell or FreeBSD's libxo?

You seem to be pretty close to Nushell, which was mentioned in the article. The Nushell `ls` (and `ps`, etc.) builtins generate structured data that can be sorted, queried, reduced, and then transformed to many different types of structured data.

$ ls | get 0 | select modified | to json

{ "modified": "2022-08-16 16:38:28 -04:00" }

The internal data format looks pretty JSON-like, with the added ability to keep Nushell types intact.

While I'm not ready to replace Fish with Nushell, it's definitely taken the place of jq for me.

What about an ioctl that lets you query what formats are accepted by the file descriptor? We already sort of have a precedent for it with giving different output based on isatty().

I dream of stdmeta for e.g. header lines:

https://unix.stackexchange.com/questions/197809/propose-addi...

I see much potential in adding stdjson as well, but I do caution against opening the floodgates to std* being implemented for every pet format and insignificant corner case.

It feels solved in raw Powershell functions, but running external CLI tools inevitably returns text and ruins the workflow.

"Crescendo" has been marketed as a solution and looks cool, but it means relearning the tool or documentation being less useful. The sheer amount of existing time people spent learning arcane git syntax means they're not going to switch to a hypothetical "New-GitCommit" function, even if it accepts arrays or PSCustomObject as input.

Is this the Crescendo you meant https://github.com/PowerShell/Crescendo ? From your comment I initially thought Crescendo was some separate commercial software.

That's the one.

It's for wrappers around existing CLI tools.

Still feels experimental, but it's a nice idea to get objects from e.g. robocopy without dropping into regex and parsing it yourself.

Cool, thanks for the info. It is very intriguing, and I admit that until this HN thread I didn't know about powershell, or appreciate this totally different model for how to connect programs together.

This is probably way off-topic now, and my question surely shows my ignorance, but do you know if there precedent for a program, once packaged to work within powershell (or maybe nushell), to use the associated input and output specification as the starting point for making a web interface to the same code (as bridged by a webserver)? Or have I just described .net ?

I'm no expert on web dev, mostly working in BI/DBA.

Running commands over WinRM or SSH can return objects of any type from remote machines. In the background I believe it's converting them to serialized CLIXML over the wire.

e.g. $RemotePSVersion = Invoke-Command -ComputerName 'SomeOtherComputer' -ScriptBlock {$PSVersionTable}

Rather than the variable $RemotePSVersion being a string it's an object with the type "System.Management.Automation.PSVersionHashTable", just like if you ran $PSVersionTable locally.

For anything that returns text (e.g. external tools like curl/robocopy) you'll usually convert to an object in your script before further processing. That way it can be passed into whatever next steps in a generic way.

e.g. curl https://catfact.ninja/fact | ConvertFrom-Json | Export-Csv '.\HighlyImportantInfo.csv'

curl https://catfact.ninja/fact | ConvertFrom-Json | Out-Gridview

That's less important when working interactively, but one major difference between Powershell/Bash is the relative focus on scripting vs interactive terminal use.

In adition to stdin and stdout, I want datain and dataout pipes, for passing some standard format.

Reply on news.ycombinator.com