Comment by jcgl | Hacker News Reader

Comment by jcgl | Hacker News Reader

Parent Full thread

I think you're mistaking text-with-structured data for structured data itself.

Because unix shell is irrevocably text-oriented, kludging in something like JSON is basically the best that can be done when you start to want to do structured operations on structured data. (I'm sympathetic to your point about the AWS CLI tools doing JSON by default though--that just sounds like bad design.)

Being text-oriented imposes drastic limits on composability. Because there is no structure, every element of a pipeline needs to do its own parsing of the input data. This leads to brittle pipelines where every element is tightly coupled to its input's textual representation.

As an exercise, try to write a pipeline that sorts podman images by size without removing the column headers[0]:

  $ podman image ls --all 
  REPOSITORY                                 TAG         IMAGE ID      CREATED       SIZE
  docker.io/prom/prometheus                  latest      937690d77350  2 months ago  367 MB
  quay.io/keycloak/keycloak                  latest      da9433c9fac3  2 months ago  466 MB
  registry.fedoraproject.org/fedora-toolbox  43          a32da54355ca  4 months ago  2.19 GB
  docker.io/powerdns/pdns-auth-49            latest      8c1385c9deed  4 months ago  208 MB
  docker.io/testcontainers/ryuk              0.13.0      b75bc7ce94c3  6 months ago  7.21 MB

As far as I can tell, there is no way to do this in a manner that's even remotely composable. Your best bet is to basically do everything from within awk. Whatever the result would be, it certainly won't be pretty!

Contrast that with what you can do in PowerShell. You can write a couple of standalone functions[0] that are readable and composable, resulting in this pipeline:

  podman image ls --all |
      Replace-SpacesWithTabs |
      ConvertFrom-Csv -Delimiter "`t" |
      Sort-Object -Property {Convert-HumanSizeToBytes -Size $_.size} -Descending

[0] Repurposing this from a blog post I wrote: https://www.cgl.sh/blog/posts/sh.html#this-should-be-basic

the point is well-taken, but i do want to show the bash version just for fun:

    podman image ls --all | sed 's/\s\s\+/\t/g' | tee >(head -n 1) >(tail -n +2 | sort -hrk 5) >/dev/null

this is _still_ all text, and we're relying heavily on sort to do a bunch of internal parsing and be in agreement with podman about how sizes should be formatted. also, for "real world" work, i dunno if the tee trick here has any kind of order guarantees, just that it works fine in this case. I'd probably just end up dropping the header and living with worse output in reality

Ooh, I was so hoping someone would take up the challenge! This is a far shorter answer than I had honestly thought was possible. More readable too, somehow? Great use of tee that I would never have come up with (though I hear what you say about there maybe not being ordering guarantees).

Unfortunately, it's not 100% correct, due to misaligned headers:

  REPOSITORY TAG IMAGE ID CREATED SIZE
  registry.fedoraproject.org/fedora-toolbox 44 5a36f433c691 2 months ago 2.14 GB
  quay.io/keycloak/keycloak latest 1361d6e49205 9 days ago 478 MB
  ...

I think that speaks to your final point, which is spot-on:

> I'd probably just end up dropping the header and living with worse output in reality

This pretty much sums up plain text and unix shell imo. It's very much the pragmatic solution here, and it's what ~100% of shell scripters would choose to do. And it should make anyone question the orthodoxy around the "power" of plain text in shells.

A variant:

  docker image ls -a | stdbuf -oL sed -r 's/\s{2,}/\t/g' | { head -n1; tail -n+2 | sort -hrk5 -t$'\t'; } | column -ts$'\t'

I used docker since that's what I have installed and I assume the output is equivalent.

sort's -t is set to tab for field separation.

stdbuf sets sed's output to only buffer a line at a time and flush, so the head in the {...} command group doesn't completely consume stdin's contents before it's passed to tail.

The column command recreates the space-aligned table based on tab-delimited input.

There is some cool stuff here.

I like using column to format the table. Appending it to alloyed's command fixes their header problem.

The stdbuf to multi-command block (term.?) is a neat trick. Although, one time when I ran this, I only got a couple lines of output. No idea why and I can't replicate it, but there could be some flakiness that results from the buffering somehow?

Question: how do the ? markers on the sort and column invocations work/what do they do?

I'm not seeing any ? marks in my code. Are you referring to the instances of $'\t'? In bash (and other shells, including recent versions of the POSIX sh specification) $'...' is treated like a C-style string complete with backslash escape sequences, so $'\t' is a way to have a tab character as a part of a command argument.

Oops, yes, I don't know why I mistyped a question mark. That's exactly what I was wondering, thanks.

It seems the sticking point is the column headers but if you as a human take them off programmatically and put them back on mentally, it's a much easier problem.

That's definitely the pragmatic choice when working with shell and what roughly everyone does. But it's is also a UX compromise that, if needed anywhere other than the Stockholm Syndrome-ridden world of unix, would be routinely derided.

Shell and plain text are too embedded in the unix-descendant world to get rid of, and I'm not advocating for anything like that. Just trying to push back against oft-repeated maxims about the power of plain text. Structured data has lots to offer for composability, power, and UX.

    $ podman image ls --all --sort=size

…or was the point more about doing it in a pipeline?

Yes, the point was about doing it in a pipeline. The pipeline is the basis for composition of plain text in the unix shell. If something as basic as sorting a table is hard to do, it should make us question just how good the unix shell/plain text philosophy actually is.

Baking --sort flags into shell tools is a sign that the tools do not compose well.

I've spent some time thinking about it, and I respectfully disagree.

I don't think a `--sort` flag is a failure of composability, the producer understands image size semantics better than the sort command, you can either bake everything into sort, making it an all encompassing command or you can simplify the previous step with a sort flag.

Your original powershell command is way more verbose and complicated, and maybe that's the price to pay for composability, but I'd rather just use the `--sort` flag and not spend hours or days coming up with some other undocumented and undiscovered way of doing it, the answer is right there ready to be used.

I think my point is that it's a violation of the UNIX principle of doing one thing really well, but I don't think that is a violation of composability.

Tl;dr: Plain text's bad composability forces the dichotomy that you identify between --sort and sort. I agree with you that --sort often can be a sensible UX choice regardless, but losing out on composable middle ground is strictly a loss in terms of power and expressiveness.

Thanks for taking some time to think about it. Despite my pretty absolute wording in GP, I do think that there's nuance here. But what I want to drive home is that the tight coupling/brittleness inherent in plain text composition systematically limits composition.

What I'm not saying is that every --sort option is bad sign for composition. Like you point at, sometimes it's just a simpler UX to have such an option included with your command. As a matter of fact, you see that in PowerShell sometimes with the -Filter option on various cmdlets.

Plain text's brittleness limits composition by promoting exactly the extremes that you point at (the extremes being overloaded sort, or overloaded producer):

> the producer understands image size semantics better than the sort command, you can either bake everything into sort, making it an all encompassing command or you can simplify the previous step with a sort flag.

I think it should be obvious that baking everything into sort is a bad idea in the general case. Like you say, the producer understands the size semantics better. Moreover, the ergonomics are lousy (see comments by users alloyed and i15e).

Baking everything into --sort is bad because it limits sorting to the producer's own predefined semantics. While arguably better than relying on sort's, the producer won't always have the semantics that the user cares about. E.g. maybe the user wants to analyze the disk space used only on a certain datastore. Maybe some datastores do transparent compression and sorting should be done by physical disk usage. And so on.

These two extremes are basically your only options in a plain text world, but structured data gives you more opportunities for composition. By moving to structured data and eliminating the need for ad-hoc parsing, users and their code can operate at a higher semantic level. In particular, the loose coupling introduced by this approach gives you access to things like lambdas. You're not alone in objecting to PowerShell's verbosity. Here's a terser version that would work if podman image output structured data rather than text, thereby kinda steelmanning this position:

  podman image ls --all |
      sort {podman size -h $_.size} -d

Basically all that has happened is that the plain text->structured parsing was dropped (again, to steelman the structured data vision) and naming conventions were made unix-y.

An alternate version if no specialized podman size command is needed and the sort cmdlet would by default look at an object's size property:

  podman image ls --all |
      sort -d

In many common cases, I think I agree with you that having --size on the producer is pragmatic and fast. A good UX choice. What's bad about plain text is that there is not/cannot be any middle ground between --sort and sort.

Now, tools shouldn't always oblige the user to compose. Composition is frequently not the best UX. But a shell that systematically limits composition is prima facie worse than one that promotes it. And plain text shells do indeed limit composition.

Sorry for the essay.

Reply on news.ycombinator.com