Comment by snprbob86 | Hacker News Reader

Comment by snprbob86 | Hacker News Reader

Parent Full thread

"piping strings is dumb"

I used to believe this. I was simultaneously learning Bash and PowerShell. I love PowerShell....(when I am stuck with Windows), but I'd never trade in my Bash shell. Piping objects is cool, but in reality, what is it really? It's just a programming language with a terse syntax for manipulating iterators.

Let's compare...

Powershell:

  Get-Process | where { $_.Handles -gt 200 }

C# 3:

  Process.GetProcesses().Where(p => p.Handles > 200)

Hmmm... what does piping objects even mean, really?

OK, so now why piping strings is interesting: forced serialization.

Every component of the transform can be debugged simply by deleting everything after that component. The new output goes to the screen or you can pipe it to a file. You get persistence for free. You can checkpoint your work and continue later. Imagine you are doing some very heavy duty processing that you only really need to do once. If you keep piping to a file along the way, you have backups and checkpoints.

What if you want to send the results to another server? You're going to have to serialize your objects. If your objects are arbitrarily complex graphs, how do you serialize them? It can get hairy, fast.

In summary: if you need to do arbitrary logic on arbitrary objects, use a programming language. If you need to transform some serialized data into some other serialized data: pipe strings.

There's probably a happy middle. Basically what you're saying is that you want a weakly-typed format, which unpacking that statement yet further means that you want a format that lets you do pretty much anything to it. Types boil down to a set of restrictions on what the data can be and what you are permitted to do with it, and a hunk o' text (TM) is just about as untyped as you can get.

But it might be nice to pass around JSON or something instead. It isn't quite as untyped, but on the grand scale of things it's not very typed.

I'm not saying that Unix's approach is the the final word on the subject. In fact, I quite agree with you: structured formats are a good idea.

I, for one, would like to see --format options added to many of the standard Unix tools. I'd probably define --format= as a shortcut for specifying both --informat= and --outformat=. It would be great to have some sort of XPath or dotted object notation on the command line rather than hacking around with awk, sed, cut, etc.

Rather than:

  ps -ef | awk -v p=$PPID 'p==$3{print $2}'

Do:

  ps --format=xml | xquery "/process[ppid=$PPID]/pid"

Or something like that. A bit more verbose, but far more understandable.

If I were to design the future, every tool would have metadata which describes which input and output formats it supports and prefers, so that most --format arguments would be optional:

  ps | xquery "/process[ppid=$PPID]/pid"

I'd prefer that 100 times over object piping.

Piping objects means that I'm dealing with objects, not string representations of objects. I'd much rather operate on a Process object than have to parse output from the ps command. For example, to find the processes whose parent is pid 987 (addressing snprbob86), I would write this on the command line in object shell:

    osh ps ^ select 'p: p.parent.pid == 987' ^ f 'p: (p.pid, p.commandline)' $

This invokes the Object Shell executable -- osh, a Python program, passing eight arguments (ps, ^, select, ...). ^ denotes piping. The select command selects processes, p, for which that parent pid is 987. discarding others. f applies a function to selected processes, p, returning tuples containing p's pid and commandline. $ prints the selected tuples to stdout.

Note that the piping is internal to the osh process, avoiding serialization to/from string between commands. That's much faster. (The initial version of Object Shell used OS piping, and spent a lot of time serializing and deserializing.)

Passing objects between servers does, of course, require serialization. So if I want to get a listing of pids and commandlines on every node of my cluster named foo:

    osh @foo [ ps ^ f 'p: (p.pid, p,commandline)' ] $

@foo looks up the definition of the cluster named foo (in ~/.oshrc or in /etc/oshrc) and runs the bracketed command on each node. The resulting (pid, commandline) tuples are serialized, sent back, deserialized and printed.

By the way, Object Shell has a Python API also. So the same remote command would be:

    #!/usr/bin/python

    from osh.api import *

    osh(remote('foo',
               [ps, f('p: (p.pid, p.commandline)')],
               out()))

Objects rule, in programs and on the command line.

Reply on news.ycombinator.com