>> The crazy think is we still are extremely bad at fitting things together - still the best way of fitting things together is the unix pipe
Is that sad or pure genius?
Is that sad or pure genius?
However it's still an interesting and thought provoking point. It makes you wonder about other ways of plugging different objects / modules together.
He's talking about the concept of fitting things together and refers to Unix pipes (in fact the example he gave was literally a string of piped POSIX commands ready for dumping into $SHELL).
My point was that this concept doesn't scale well.
As a productivity tool, pipes are invaluable. But you wouldn't want to write a performance critical routine using them.
At the end of the day (and as Joe said himself), it's about picking the right tool for the job.
I can't really see much difference between grep | sed |awk somestuff.txt and (awk(sed(grep(somestuff.txt))).
Or are you suggesting that function composition is not the right approach? In which case, I disagree, but I would like some more insight into your thought process on this.
1) You've only got one send and one return. (well, arguably more if you include stderr and exit codes, but they have their own limitations in addition to the aforementioned)
2) They're too insular from each other; different files dotted about that needs to be loaded into the memory then executed. It adds quite a massive overhead. Granted this isn't an issue for the kind of jobs you'd write shell scripts for in the first place, but it does severely hamper this kind of model in terms of scalability. But then we're back to the age old issue of performance vs convenience.
I will concede that the second point is more an issue about implementation rather than concept though. But if you were to write a lower level implementation of Unix pipes, I don't think it would pretty much end up with Perl functions (which, to be fair, you suggested yourself). I say Perl specifically because you can create a function without specifying what values to accept (see example below), which is akin to the 'dumb' strings read in from STDIN.
sub PerlFunc() {
foreach (@_) {
$i++;
print "Parameter $i == $_\n";
}
}
I'm not really sure if that answers your question though (or even if I've said anything you don't already know).
I think GP was confused about the difference between the concept of piping and the concept of function composition. As noted by GGGP above, the concept of piping/composition is not only Unix pipes, but you keep arguing about that particular case of it instead of the general concept.
Function composition really isn't that uncommon.
Case in point: no sane person would rewrite Apache in Bash. But equally I wouldn't rewrite any of my sys admin shell scripts in C.
There's a balance that needs to be struck. Which I did also hint at in my previous post (the one you dismissed outright when arguing about 'unhealthy concentrations'). And I think what Joe actually meant by that quote was the same thing; that many developers are bad at judging that balance.
So if I am reading too much into your post, then what was the intent?
However, my goal was to see more about where you draw the line, and I think you did answer that, so thanks.
Thank you for the clarification :)
You end up with every tool containing code to parse a stream of bytes into the appropriate data structures, and then probably also write them out to a stream of bytes too. The user has responsibility for a lot of data munging too. It is error prone and repetitive, at least when dealing with anything other than very simple text files containing lines of strings where you know which encoding has been used. (Re)using libraries that read and write more structured data is likely to get you flamed[1]. The history of Unix contains plenty of security problems due to people not correctly accounting for nulls or control characters or spaces etc when chaining together commands (although this is more of a problem with shells than pipes themselves). The output is often not friendly to human eyes by default (I'm thinking of things like localised date and time formats, number formats and so on) since it has to be suitable for passing to another program, unless you use more options or another tool to reformat it.
I would compare it to working with raw memory in C. It's the lowest common denominator, you can do anything, but it's also trivial to screw up and there is a high cognitive overhead. Perhaps it's just too hard to introduce anything higher level at this point.
Perhaps my opinion would change if I spent years (deeply) learning unix tools, but maybe that would just be because I had invested so much time learning a complicated system.
Granted not quite the same thing, but raw bits are often used in lower level languages. Take Windows Win32 APIs, there's a lot of instances where styles are defined by adding constants with values being exponentials of 2. Thus creating a binary array of boolean states.
Another example I used to run into was Windows' controller (joystick et al) API (again Win32). Each bit would represent a different controller button and the 'on' state was if the button was depressed. But as the value was returned as an unsigned long int (if memory serves), it was up to the developer to write their own parser to convert what would otherwise been a random number into a meaningful array of bits. (or at least I did - there is a chance I overlooked another function as this was before I made the switch to DirectX6 - so many years ago!)
http://theatticlight.net/posts/Why-cant-we-do-pipes-smarter/