Rob Pike: "Current Unix tools are weakened by the built-in concept of a line"

recordstream seems to concentrate on lots of formats. Really, it should concentrate on lots of tools, with a shared repository of parsers for tools.

EDIT: Actually, they concentrate on JSON, but they also try to provide a generic set of tools for reading any format. I think this could be better structured to "just work."

First thing I checked for too. Second thing I looked for was someone pointing out that this is asking for object oriented systems (such as Smalltalk or Self).

But the third thing, no one mentioned: SNOBOL. Reportedly (because I haven't used it myself) it is better than AWK for complex matching.

Allow a shell variable to control output record and field separators. Default to space and newline if nothing is specified.

ORS=: OFS=, ls -l # rw-r--r--:1:uname:gname:1025:Jun 1:somefile.txt,...,...

Add a specifier for dates, usernames, groupnames, etc. DSF="%Y-%m-%d" ls -l

When I talk about a "representation," I mean an in-memory format. For example, the "representation" of an HTML tree is the DOM.

Yes, you have to agree on a serialization format (JSON, Protocol Buffers, etc), but that's not the same thing. From a serialization format you can represent the data however you see fit in your process. For example, a C++ user might represent a string as a std::string object whereas a Python user would represent it as a native Python string.

The VM-based approach (like PowerShell) defines an in-memory tree representation, namely .NET objects. This means that you can't really interoperate with this stack unless you use .NET too, since you don't have an easy way of converting .NET objects to your own objects.

You are confusing the "how" (the API specification) with the "what" (the data the API exists to provide access to).

So things like CSV, XML, and JSON are APIs now?

His point is that he's pushing off the extra hard work to be done only once, rather than at every pipe. I'm not sure how he needs to recheck his assumptions there since he stated them.

"ls" prints a list of files. "find" knows about the filesystem and can print a list of files based on file metadata and a path (such as a filename). If you have a list of files (eg from "ls") -- you would need to look up the metadata you want to filter on; it is not part of ls' interface to give them to you directly:

  ls | find -size +1M

On the other hand, if you want to make list of files and their sizes, that can be stored, sent over the network, etc -- and then filer that list, you can do:

  ls --size | awk 'file_blocks=$1 $file_blocks > 1024 { print $2 }'

For most tasks, I think the fact that any human can look at the output from eg: "ls --size" and then produce a valid test dataset in the same format is more valuable than having to explicitly "cast" the metadata while processing.

>>Let me ask you this; how would you do the equivalent of this hypothetical command? >> $ ls | structured-grep 'file.size > 1M'

# find . -maxdepth 1 -size +1M

really, flat text is fine. Maybe it's not perfect, but it's good enough that most want to not add any complexity to it that would make it incompatible.

And if you really need that complexity, it's usually worth whipping up a parser for.

The solution I often contemplate is to address the process by which the data is created. (Let's assume we're talking about data you would find on the web.) That is, why don't we impose rules on that process? Why can't we "mandate" that the output be structured data from the get go? Instead we allow vast quantities of unstructered data to be created and then we try to normalize it. I know this is a radical view, but it could make sense in some circumstances.

I disagree with your comment about CSV. But it's impossible to have a meaningful argument unless you provide an example: Give me a job to do, a CSV file and let me have a go at it. I'm serious. Post a link to a CSV file, define a task and let's see what we can do just using plain ole UNIX. Could be a fun exercise.

As for your hypothetical command, I do not understand what is so difficult about this. The stat command is what you want, not ls. No self-respecting UNIX user would parse ls when he can use stat (I recommend the BSD one over GNU.).

But here's what I would do:

1. If your UNIX filenames have spaces in them, rename them. There is no sensible reason to leave spaces in filenames in UNIX. Fix this first before it becomes a problem.

2. Write a one-liner and save it as a function, perhaps in your .profile, or maybe in RCS, or save it as a script. There's so many ways to manipulate output as a stream. Pick one that suits your tastes. That's the beauty of UNIX. Make your own solutions as you go. There is no right or wrong answer. It is a form of customization. My choice will no doubt make some people cringe. Assuming there's no user named "[0-9]M":

   whatever(){ ls -lhS |tr '\011' '\040'|sed '/ [0-9]M /!d' ;}

or save what's between the brackets as a file named "whatever". Maybe you save it in a directory called "x" and add that to your PATH. Then you do

   . whatever

Of course how long the list of files is going to be makes a difference. I might take a different approach if the output was going to be an enormous list.

There are so many ways to get what you're after. The point is that you should be able to tap out a one-liner that does the job. Maybe it takes a few iterations to get the right output. Tweak it until the output is what you want. Viewing command-line history is perfect for seeing the process of creating a one-liner to manipuate output. You can see the line grow incrementally as you build it, until you finally have the output you want. This sort of history allows you to go back to any stage in the process. If you're a vi fan, you can use vi-mode on the command line to move around the line quickly as you edit. Eventually you can hit "v" and edit the thing visually in your EDITOR, then save it. I've built over 700 useful functions this way and the number keeps growing.

I do understand there should be a way to "extract" the file size column the way Pike decribes in the article. To do this I think you have to free yourself from "line-oriented" thinking and imagine another type of structure. And I think using another language you can do it. But for something as simple as this -- manipulating ls command output (cf. manipulating large datasets) -- an "ugly" one-liner suits me fine. The more you use the boring old utilities the more you can get them to do.

Regular Expressions are indeed "crude". But, to me, that is just fine in a lot of cases.

Acme mouse chording is really cool, doing something similar with keyboard-oriented editors is much more tedious and often outright painful ( http://acme.cat-v.org/mouse ).

I suspect ken spends most of his time thinking rather than typing, and I have found this to be true of most great hackers.

Other famous Sam users include Brian Kernighan, Bjarne Stroustrup(!) and Tom Duff. Kernighan writes at least as much English as code and I wonder how that affects his editor usage patterns, but I suspect that even when writing natural languages most time is best spent thinking (his writing style is very concise and clear, one could say similar to ken's code).

I think the obsession with saving keystrokes is very misguided, I still use editors like vi frequently, and having to think about what magic combination of commands to use to perform a task can be very distracting, is fun and feels good, like a tiny puzzle game built into your editor, but it doesn't help you write better code faster IMHO.

P.S.: Is interesting who has stuck with Sam and who moved to Acme (Dennis Ritchie switched to Acme, and most of the Go team Google besides ken use it too).

See also: http://sam.cat-v.org/

continuing the offtopic:

You say it sounds like an interesting method at first, but then you try it out and it's weird and slow. How much of a try did you give these editors? In the same way that a new Vim user will basically use the arrow keys, "i", ":w", and ":q" and nothing else, a new user to Sam or Acme can very easily miss a lot of the power.

The Sam language is pretty powerful. For instance, you can use the "X/regexp/ command" form to apply a command over every file whose filename matches the regex, so you can (for instance) make a change to every .c file while leaving README alone. Acme lets you use the same command language, but also lets you execute other arbitrary commands by simply typing them and mid-clicking on them--yes, I know Vi lets you do something similar, but with Acme you'll typically build up a "guide" file, full of convenient commands that you just sweep over with the middle mouse button and release to execute. Acme also presents files in a sort of tiling window manager fashion that makes it one of the most convenient editors I've used. I'll frequently have up to about 50 files open, which in Emacs or Vi would drive me nuts trying to constantly switch around them or split the screen into one or two panels (oh boy, C-x b!). In Acme, I can always see the titles of the individual buffers, and if I need to see into the file I can with a mouse click or two expand the buffer into a convenient size.

I think one change that could really have a big impact is making the ESC key switch back to the command window; this makes things a lot more familiar for Vi-heads, and it makes sense to eliminate a mouse movement for something that simple and frequent.

That ended up a lot longer than I intended, but I wanted to try and share a little bit of my thoughts on Acme and Sam, and encourage others to give them a more thorough try. At this point, I use vi for quick edits, but when I have to write a lot or make a lot of changes, I bring out Acme. When I'm stuck on a strange system that's not my own, I have a tarball of an old version of Sam which, with a little tweaking, has compiled on every version of Unix I've had to use.

>Okay, thanks, but I've known about basic automata theory since I was an undergrad, two decades ago.

I wasn't trying to educate; I was discussing the relevant limitations of my approach. The fact that I can't spin a perfect regexp for anything (including JSON, sexp, xml) that nests arbitrarily deeply is an issue with what I proposed - one that I think can be worked around sufficiently, but an issue nonetheless, and I wanted to acknowledge that. I don't see what prompted the defensiveness I take from your comment - I'd expect most people here to know at least that much automata theory, here. I've not known it for quite two decades, but two decades ago I was 8. I'll reply to the constructive bits of your response separately.

It appears that I did misunderstand your suggestion - I'd thought you to be proposing a separate set of tools (grep, sed, etc) for operating on each packet format. Instead, it appears you meant a set of tools for converting output to a particular format? That still means either 1) reimplementing tools to deal with each format, or 2) many conversions and perhaps still an inability to get the data to chunk the desired way for any particular tool. Just the ability to have sed, grep, and sort chunk in an arbitrary way would be significant.