Hacker News Reader

Comment by zokier | Hacker News Reader

What sort of features do you feel that shell needs to support structured data? I've toyed with this idea, and in my view the shell itself has relatively minor role here. What is needed are utilities that produce and process structured data, and a terminal that can display it. But shell, it just glues the pieces together and doesn't really need to be aware of the data and the structure of it.

View on HN

Here's a very simple example that I'd consider the holy grail. Say I want run `ls -lh` on a directory:

    -rw-r--r--   1 brandur  staff   6.5K Mar 25 15:37 Makefile
    -rw-r--r--   1 brandur  staff    63B Jul 10  2016 Procfile
    -rw-r--r--   1 brandur  staff   1.6K Mar 11 07:20 README.md
    drwxr-xr-x   4 brandur  staff   136B Oct 16  2016 assets/
    drwxr-xr-x   4 brandur  staff   136B Jul 15  2016 atom/
    drwxr-xr-x   4 brandur  staff   136B Apr 26  2016 cmd/

I'm interested in getting the date fields out of this structure. Currently that's possible with some disgusting use of `awk` or maybe `cut`.

What if I could do something like `ls -lh | select 6` (select the 6th column of data):

    Mar 25 15:37
    Jul 10  2016
    Mar 11 07:20
    Oct 16  2016
    Jul 15  2016
    Apr 26  2016

It doesn't matter that the date comes in three parts and would act as three separate tokens for `awk` and `cut` because in my structured object backend, it's all just one logical timestamp.

Now say I want to sort these dates and pick the earliest. Once again, shells make this extremely difficult. Because the dates are not lexigraphically orderable, I'd have to use some fancy parsing, or try to step back to `ls` and have it somehow order by date.

Imagine if I could instead to something like `ls -lh | select 6 | sort | first`:

    Apr 26  2016

In this case, although the shell is still displaying a string to me that's not lexigraphically orderable, the data is being passed around in the backend in a structured way and these timestamps are actually real timestamps. It's then trivial for the next utility to apply ordering.

The part that the shell would have to provide (as opposed to the programs) is a "smart pipe" that understands more than just a basic text stream. It would also have to know when there is no pipe connected, and print as pretty strings when it knows the output is going to `STDOUT`.

Not quite a "Unix" shell, but the [Ammonite Scala Shell](http://ammonite.io/#Ammonite-Shell) lets you do this trivially:

    lihaoyi Ammonite$ amm
    @ import ammonite.ops._
    import ammonite.ops._
    @ ls! pwd
    res1: LsSeq =
    ".git"              'LICENSE            'ci                 'project            'sshd
    ".gitignore"        'amm                'integration        'readme             'target
    ".idea"             "appveyor.yml"      "internals-docs"    "readme.md"         'terminal
    ".travis.yml"       "build.sbt"         'ops                'shell
    
    @ ls! pwd | (_.mtime)
    res2: Seq[java.nio.file.attribute.FileTime] = List(
      2017-06-21T12:24:41Z,
      2017-06-11T13:48:45Z,
      2017-06-21T12:29:05Z,
      2017-06-11T13:48:45Z,
      2017-05-01T03:41:14Z,
      2017-06-18T14:06:33Z,
      2017-06-11T13:48:45Z,
      2017-06-18T08:01:20Z,
      2017-06-18T12:34:00Z,
      2017-06-19T06:05:51Z,
      2017-06-18T09:06:07Z,
      2017-06-18T14:06:33Z,
      2017-06-18T14:06:15Z,
      2017-06-18T14:07:20Z,
      2017-06-11T13:48:45Z,
      2017-06-18T14:06:33Z,
      2017-06-18T14:06:33Z,
      2017-06-19T06:06:03Z,
      2017-06-18T14:06:33Z
    )
    @ ls! pwd | (_.mtime) sorted
    res3: Seq[java.nio.file.attribute.FileTime] = List(
      2017-05-01T03:41:14Z,
      2017-06-11T13:48:45Z,
      2017-06-11T13:48:45Z,
      2017-06-11T13:48:45Z,
      2017-06-11T13:48:45Z,
      2017-06-18T08:01:20Z,
      2017-06-18T09:06:07Z,
      2017-06-18T12:34:00Z,
      2017-06-18T14:06:15Z,
      2017-06-18T14:06:33Z,
      2017-06-18T14:06:33Z,
      2017-06-18T14:06:33Z,
      2017-06-18T14:06:33Z,
      2017-06-18T14:06:33Z,
      2017-06-18T14:07:20Z,
      2017-06-19T06:05:51Z,
      2017-06-19T06:06:03Z,
      2017-06-21T12:24:41Z,
      2017-06-21T12:29:05Z
    )
    @ ls! pwd | (_.mtime) min
    res4: java.nio.file.attribute.FileTime = 2017-05-01T03:41:14Z
    @ ls! pwd | (_.mtime) max
    res5: java.nio.file.attribute.FileTime = 2017-06-21T12:29:05Z

    @ ls! pwd maxBy (_.mtime)
    res6: Path = root/'Users/'lihaoyi/'Dropbox/'Github/'Ammonite/".idea"

Ammonite certainly has problems around JVM memory usage and startup times, but it has a sweet spot for this sort of not-quite-trivial filesystem operations that are a pain to do in Bash but too small to be worth futzing with a Python script

Doesn't ammonite suffer from the same "problem" as powershell that it is limited to single runtime (namely JVM) and single process? In contrast in traditional Unixy pipelines each component runs in its own process and can be programmed in any language.

You can also do it in PowerShell, which is on linux now.

  PS C:\Users\erk> Get-ChildItem | Sort-Object LastWriteTime | Select-Object -first 1
  
  
      Directory: C:\Users\erk
  
  
  Mode                LastWriteTime         Length Name
  ----                -------------         ------ ----
  d-----       17-06-2016     16:08                Tracing

Part of me thinks that PowerShell has seen less adoption in the Unix world because of the prevalence of camel case. The few times I used PowerShell, the discoverability of what I could do was low and the verbosity of examples was high. It took me too long to do my most common tasks.

I've seen several people who like PS say its discoverability is actually pretty great compared to Unix shells - for instance, the convention for cmdlet names is always "verb-noun" ("action-object") and (maybe, don't quote me on this) the set of blessed verbs people are supposed to use in cmdlet names is fairly small. So, when you want to do something, you try the verbs that seem closest to what you want to do and the names of the (usually application specific) objects you're working with and that gets you quite a long way down the road.

Also I think PS commands aren't case sensitive, though for readability one would probably want to keep the camelcase anyway.

Yeah PowerShell is not case sensitive, and it also have a lot of unix-like aliases you can use, Another way to write the command of post is:

  dir | sort LastWriteTime | select -first 1

It gives the same output.

But if we have a program that produces structured data ("ls"/"get-childitem") and a program that similarly consumes structured data ("select-column"), then I'm not sure what sort of "smartness" you would need from the pipe in between? Why would the shell need to understand that the data is somehow more than just stream of bytes?

As far as pretty-printing the final output to the user, in my opinion that is something that is better handled at terminal level instead of the shell.

Of course these are just comments based on how I imagine a reasonable object shell would function, and I really would like to hear more views on this subject because it is very likely that I might have overlooked something essential.

'ls -lh | column --table --output-separator "," | cut --delimiter="," --fields=6-8 --output-delimiter=" " | sort --month-sort'

I statred this to show how easy it was, but it quickly became pretty hacky.

Your point is taken, but in this particular example, you should just use the right tool for the job. In this case, the "stat" utility. For example:

  > stat --format='%z' *
  2016-05-27 12:39:46.559137232 -0300
  2015-07-26 14:37:51.714193856 -0300
  2016-04-15 19:55:33.329346654 -0300
  2016-03-21 02:59:05.377041620 -0300
  2015-11-22 19:27:56.868541801 -0300

Sorting them and picking the earliest would just be:

  > stat --format='%z' * | sort -n | head -n 1
  2015-07-26 14:37:51.714193856 -0700

Powershell?

Reply on news.ycombinator.com