Hacker News Reader

Comment by Mister_Snuggles | Hacker News Reader

In FreeBSD, this problem was solved with libxo[0]:

    $ ps --libxo=json | jq
    {
      "process-information": {
        "process": [
          {
            "pid": "41389",
            "terminal-name": "0 ",
            "state": "Is",
            "cpu-time": "0:00.01",
            "command": "-bash (bash)"
          },
    [...]

It's not perfect though. ls had support, but it was removed for reasons[1]. It's not supported by all of the utilities, etc.

This seems to be a great stop-gap with parsers for a LOT of different commands, but it relies on parsing text output that's not necessarily designed to be parsed. It would be nice if utilities coalesced around a common flag to emit structured output.

In PowerShell, structured output is the default and it seems to work very well. This is probably too far for Unix/Linux, but a standard "--json" flag would go a long way to getting the same benefits.

[0] https://wiki.freebsd.org/LibXo

[1] https://reviews.freebsd.org/D13959

View on HN

> In PowerShell, structured output is the default and it seems to work very well.

PowerShell goes a step beyond JSON, by supporting actual mutable objects. So instead of just passing through structured data, you effectively pass around opaque objects that allow you to go back to earlier pipeline stages, and invoke methods, if I understand correctly: https://learn.microsoft.com/en-us/powershell/module/microsof....

I'm rather fond of wrappers like jc and libxo, and experimental shells like https://www.nushell.sh/. These still focus on passing data, not objects with executable methods. On some level, I find this comfortable: Structured data still feels pretty Unix-like, if that makes sense? If I want actual objects, then it's probably time to fire up Python or Ruby.

Knowing when to switch from a shell script to a full-fledged programming language is important, even if your shell is basically awesome and has good programming features.

Are executable methods really that bad? I mean, they're bad in some abstract sense but that seems more like an objection if we were talking about a "safe" language like Rust than talking about shell scripting. For a shell executable methods seem fine. If you don't make the method executable people are just going to use eval() anyway, might as well do the more predictable thing.

It might be possible to design a good Unix shell based on objects, with the ability to "call back into" programs. But I haven't seen one yet that I'd prefer over Ruby or Python.

I do think objects make plenty of sense in languages like AppleScript, which essentially allowed users to script running GUI applications. And similarly, Powershell's objects might be right for Windows.

But nushell shows how far you can push "dumb" structured data. And it still feels "Unix-like", or at least "alternate universe Unix-like."

The other reason I'm suspicious of objects in shells is that shell pilelines are technically async coroutines operating over streams! That's already much further into the world of Haskell or async Rust than many people realize. And so allowing "downstream" portions of a pipeline to call back into "upstream" running programs and to randomly change things introduces all kinds of potential async bugs.

If you're going to have a async coroutines operating on streams, then having immutable data is often a good choice. Traditional Unix shells do exactly this. Nushell does it, too, but it replaces plain text with structured data.

PowerShell is basically an interactive C# shell with language ergonomics targeting actual shell usage instead of "you can use it as a shell" the way Python, Ruby, etc. approach their interactive shells. However, the language and built-in utilities work best when you are passing around data as opposed to using PowerShell as if you were writing C#.

It's true, you are indeed passing around full-blown .NET runtime objects. In fact your whole shell is running inside an instance of the .NET runtime, including the ability to dynamically load .NET DLL's and even directly invoke native API's.

It feels a bit like JS in the sense that you're best off sticking to "the good parts", where you get the power of structured input/output but you don't end up trying to, for example, implement high performance async code, even though you technically could.

> In PowerShell, structured output is the default and it seems to work very well. This is probably too far for Unix/Linux, but a standard "--json" flag would go a long way to getting the same benefits.

OP has a blog post[0] which describes exactly this. `jc` is described as a tool to fill this role "in the meantime" -- my reading is that it's intended to serve as a stepping stone towards widespread `-j`/`--json` support across unix tools.

[0] https://blog.kellybrazil.com/2019/11/26/bringing-the-unix-ph...

If there is to be a push to add structured data to all the unix tools, I wish they would use a format that allows embedding binary data. Yes base64 is an option, but it suffers from the issue that base64(base64(base64(data))) leads to exponential overhead.

CBOR? It's based on JSON, so it should be pretty easy to add to a tool that already has JSON serialization.

Similarly, in SerentityOS, stuff under /proc return JSON data rather than unstructured text files.

A better, structural way in which this could be fixed is to allow data structures to be exported in ELFs and have those data structures serialized into terminal output, which can then be outputted in the preferred format of the user, such as JSON, YAML, or processed accordingly.

I mean if you have a filesystem you've already got a way to tree-structure your data...

Libxo is neat, in theory, but it seems like applications are left to implement their own logic for a given output format rather than being able to pass a structure to libxo and let it do the formatting.

I can't remember the exact utility--I think it was iostat--would use string interpolation to format output lines in JSON and combined with certains flags produced completely mangled output. Not sure if things have improved but I would have expected something like JSON lines when interval is provided.

Powershell and kubectl are miles ahead of libxo in useability imo

Well I suspect that eventually you just run into hard limitations with C's introspection facilities, or lack thereof.

I like C a lot but one of the reasons I like Rust more these days is the ability to trivially implement complex serialization schemes without a ton of ad-hoc code and boilerplate.

far better for applications to be unaware of a such a utility and allow something like jc to grow in support with plugins or something so as to keep the utilities simple and move the logic and burden to the wrapping utility in this case jc.

> In FreeBSD, this problem was solved with libxo[0]:

Libxo happens to be in the base system, but it is generally available:

* https://github.com/Juniper/libxo

* https://libxo.readthedocs.io/en/latest/

What I find weird about Powershell is that there's no "fixed-width column" parser, which is a widely used format for Unix-style CLI tools.

I don't know if NuShell has it, I haven't tried.

In any case, it's much better for tools to output more-parseable data in the first place. Whitespace-delimited columns are fine of course, but not so much when the data can contain whitespace, as in the output from `ps`.

I don't see much reason why JSONLines (https://jsonlines.org/) / NDJSON (https://ndjson.org/) can't be a standard output format from most tools, in addition to tables.

As for the reason of removal:

  any language that can interact with json output can use readdir(3) and stat(2).

Ugh. Any language of course can do it. But that's basically telling users that they need to reimplement ls(1) themselves if they want to use any of its output and features in scripts.

I understand if the maintenance burden is too high to put it in ls(1) itself, but it's a shame that no tool currently does this. The closest we have is a feature request in Eza: https://github.com/eza-community/eza/issues/472

> This seems to be a great stop-gap with parsers for a LOT of different commands, but it relies on parsing text output that's not necessarily designed to be parsed

True, and yet it's extremely common to parse output in bash scripts and other automations, so in a sense it's just centralizing that effort. That being said at least when you do it yourself you can fix problems directly.

Now they only need to do the same thing for input and let the operating system or the shell handle the argument parsing so that it is consistent accross the entire operating system.

> It's not perfect though. ls had support, but it was removed for reasons

In specific:

https://svnweb.freebsd.org/base?view=revision&revision=32810...

> libxo imposes a large burden on system utilities. In the case of ls, that burden is difficult to justify -- any language that can interact with json output can use readdir(3) and stat(2).

Which rather misses the point of being able to use JSON in shell scripts.

I'd love to know what the burden was too. I hear comments like that in code reviews and commonly when you push for specifics about the burden, there is very little

Reply on news.ycombinator.com