I always hated that part of UNIX. It would be so much better if programs could handle data-structure streams instead. Having text streams causes every single program to implement ad-hoc parsers and serializers.
I now use the command-line only for trivial one-time actions. For everything else I'll use a scripting language and forget about reading and writing text streams altogether.
Text is a useful common denominator. Text is possible to version control, tie to bug trackers, and handle with configuration management systems.
The same is true for the command line. If you handle structured data, or objects, you communicate using APIs. While it's not theoretically impossible to still use version control and configuration management, it turns out that it's much more difficult in practice. Plain text is a useful lowest common denominator.
I would much rather have functional primitives (map, filter, reduce, zip, take, drop, etc) doing this work.
It's like the difference between static and dynamic typing. Solving the type system's constraints adds complexity over and above the irreducible complexity of the problem. Static typing pays for its added complexity by proving certain things about the code, but for ad-hoc, short-lived code it usually isn't worth it. And most code (by frequency, if not by importance) using streams is ad-hoc, on the command line.
With a structured stream, there are only a handful of generic utilities that make sense: map, filter, reduce, etc. (and they better have a good lambda syntax). Whereas the advantage of unstructured streams is that utilities that were never designed to work together can be made to do so, usually with relatively little effort.
For example, suppose you have a bunch of pid files in a directory, and you want to send all the associated processes a signal. What kind of data structure stream does your signal sending program accept? What needs to be done to a bare number to convert it into the correct format? How do you re-structure plain text from individual files? Structure in streams seems to have suddenly added a whole lot of complexity and work, and for what?
Whereas:
cat $pid_directory/*.pid | xargs kill -USR1
What program? A single line of shell code would work fine. Kill itself only need take a pid, or an actual handle if Unix had sich a thing.
>What needs to be done to a bare number to convert it into the correct format?
If a "bare number" isn't the correct format, why would you have them at all?
>How do you re-structure plain text from individual files?
The whole idea is not to use plain text at all.
>Structure in streams seems to have suddenly added a whole lot of complexity and work, and for what?
Structuring your data doesn't add complexity; when you consider the hoops one jumps through to strip data of its structure at one end of a stream and reconstitute it at the other, it's really reducing it. It's only if you insist on also using unstructured representations that complexity is increased.
Of course, as long as Unixes and their shells only speak bytestreams and leave all structuring, even of program arguments, to individual programs, it's a moot point. He's still right aboutnit being a shitty design, though.
Exactly this. I think HN doesn't have much experience with powershell, which is why you're currently being downvoted. So let's have a practical example: consider killing processes run in the last 5 mins using posh:
ps someprocess | where { $_.StartTime -and (Get-Date).AddMinutes(-5) -gt $_.StartTime } | kill
Now try the same on bash, and spend time creating a regex to extract time from the unstructured output of Unix ps.
Not really a complex regexp thogh. I almost exclusively use Linux and thus bash/zsh etc. And yes, my piece above looks uglier and like but of a hack, but that's not the point. It's easy because it's discoverable. These are one-liners you write ad hoc and use once. But powershell in my experience lacks the discoverability that bash has, you can't look at what some tool outputs and then know how to parse it, you need to look at the underlying object first. Granted I have maybe one day of experience with PowerShell, but I don't know anyone who uses it for their primary interaction with the Computer. For Bash though...
(And yes I'm aware that you can also create huge complicated bash scripts, but you could also just use python)
Find the name of the CPU using powershell and have fun looking up the correct WMI class and what property you need.
Here's bash: grep name /proc/cpuinfo
Well it was precisely my point.
get-wmiinfo seems pretty discoverable to me. You can browse the output and pick what you want.
Being able to stream a collections of bytes (and collections of collections of bytes, recursively) is one case that I find myself wanting when sending data between programs at the command line.
Consider:
ls "$pid_directory" | xargs rm
If there were some way to encode items in a collection OOB, you could pipe data through programs while getting some guarantees about it being represented correctly in the recipient program. (Sometimes you see scripts that do this by separating data in a stream with NUL delimiters, but this doesn't work recursively or if your main data stream might have NUL in it.)
If you don't mind using JSON as intermediary format, you might like to have a look at jq: http://stedolan.github.io/jq/
jq can also convert raw (binary) data to JSON objects containing strings (and vice versa) for further processing. Naturally, jq filters can be pipelined in many useful ways.
If programs passed data structures then either you're forcing a certain data structure model (i.e. it's not universal, because it's not compatible with anything else), or your data structures are so general (i.e. a block stream) that your applications are going to be parsing anyway... and that's going to be even nastier than if everything was stupid text steams in the first place.
For this to work data structures would have to be nothing more than scalars, sequences and mappings without specific concrete types. Just like JSON, YAML, and the rest do it now.
That's fine for personal use, or in a single project, but doesn't scale like "dumb" text does.
It would be easy to wrap an arbitrary other packed format inside a binary string, or an arbitrary text format inside a text string.
IFF is quite successful in certain areas, BTW.
[1]: http://en.wikipedia.org/wiki/Interchange_File_Format [2]: http://en.wikipedia.org/wiki/BSON
Of course, you could also do something similar with a 0mq adapter as well for a request/response server, which is available pretty much everywhere you might need it (language wise)... It's an interesting generic interface... however as I mostly use node for newer development, it's so trivial to use sockets or websockets directly for this, I'd just assume go that route... or for that matter, something like sockjs/shoe.
At the same time, if you want to write a utility like grep that is agnostic to the structure of your text, it can exist and work. If UNIX cared about data structures, this wouldn't work.
Here's a good rule for you; if you find yourself thinking the way UNIX (or any other long existing and widely supported system) does something is dumb and you know better, assume you are wrong and look for the reasons that the people who are smarter than you chose to do it the way that it was done.
While there are some occasional good points that could be converted in to solutions, I can't help but think that this book is more amusement than actual constructive criticisms well packaged.
My point was that the parent's suggestion that anyone who criticizes Unix obviously isn't smart enough to understand it is just flat out wrong and offensive.
That wasn't what I was saying at all. But thanks for playing.
So even within a controlled environment like PowerShell where everything is a cmdlet it is still impossible to pass only structured objects between commands.
In other words you still haven't solved the fundamental problem of passing data back and forth between different programs. In fact if you are going to mandate a specific serialization/deserialization format then JSON, XML, or even ASN.1 are better options than s-expressions.
The other point was that we're ultimately stuck with serial forms of communication, be it wires, pipes, sockets etc. If we want to easily transfer structured data through these serial channels, we should probably build up our structures from a serial primitives, and S-expressions are much more handy than plain strings (which we may not even be able to parse without ambiguity), or XML, JSON or whatnot. One, because the parser is already implemented as part of the language, and secondly, because you can transfer code in addition to data, and evaluate it on the remote end to bring into scope more "structured" data like records.
I did try to include a bit more in the previous post, but I'd accidentally hit save, and I was unable to edit the post afterwards
The advantage of using a proper data format is
a) You don't have to do in-band signalling so it will be far more reliable (you still can't have spaces in filenames for a lot of unixy things). b) The encoding is standard. Using text for pipes still requires some kind of encoding in general, but there are many different ways (is it one entry per line? space separated? are strings quoted? etc.)
When someone has example of situation when binary, json or other communication between websocketd and program is needed, please just file a ticket, it would be great to see practical situation instead of just arguing with each other about text stream/unix principles/json and other stuff.
It's something to ruminate upon.
There's also msgpack, protocol buffers as well... I think the plain text of json that is readily a line per message is far simpler and easier to handle though.
All you really need are scalars, sequences and mappings to fully express any data structure. Tagged types would be sugar on top but it could quickly add unwanted complexity.
All you need is a shell capable of marshalling data between the different programs piped together. Powershell is a nice idea but it only runs .NET code and that's a limitation I can't live with.