Relational shell programming

There was an entire system called "rdb" written around this philosophy of flat-files-as-relations:

http://en.scientificcommons.org/43495509

I'd be curious if anyone actually used it -- is it obscure because it is a bad idea or because it didn't offer much more over the shell, etc.?

"There are many database systems available for UNIX. But almost all are software prisons that you must get into and leave the power of UNIX behind. Most were developed on operating systems other than UNIX. Consequently their developers had very few software features to build upon, and wrote the functionality they needed directly, without regard for the features provided by the operating system. The resulting database systems are large, complex programs which degrade total system performance, especially when they are run in a multi-user environment. UNIX provides hundreds of programs that can be piped together to easily perform almost any function imaginable. Nothing comes close to providing the functions that come standard with UNIX. Programs and philosophies carried over from other systems put walls between the user and UNIX, and the power of UNIX is thrown away. The shell, extended with a few relational operators, is the fourth generation language most appropriate to the UNIX environment. 1. Fourth Generation Systems In recent years, a variety of developments in programming language design have emerged. Object-oriented languages are becoming common, and languages explicitly supporting multiple tasks and inter-task communication are also gaining popularity. "

If you're a student in mattmight's class, you shouldn't read this.

N hfrshy gbby gung negvpyr zvffrf vf gung frg qvssrerapr pna or vzcyrzragrq (zhpu zber rssvpvragyl guna uvf dhnqengvp nytbevguz), hfvat naq .

Vs lbh unir gjb svyrf N naq O juvpu ner frgf (v.r., ab vagreany qhcyvpngrf), N - O pna or rkcerffrq ol "fbeg N O O | havd -h".

Bs pbhefr, vs lbh TAH pberhgvyf ba lbhe flfgrz (naq vs lbh'er ehaavat Yvahk, lbh nyzbfg pregnvayl qb), vg'f rira rnfvre naq zber rssvpvrag: "pbzz -23 N O".

The author misses the key command for performing set operations:

    comm

This can perform set differences, unions, and even symmetric difference.

    comm -23 a b   # a minus b
    comm -12 a b   # a and b
    comm -3  a b   # a symmetric difference b (with some extra tabs)

A more idiomatic way to phrase the awk in the Selection example:

    awk -F ":" '{ if ( $3 != $4 ) print }' /etc/passwd

would be like this:

    awk -F ":" '$3 != $4 { print }' /etc/passwd

The first is obviously valid awk, but the second highlights awk's unique "pattern { action }" syntax.

join is also worth mentioning: as expected, performs relational join of supplied input, equivalent to SQL's STRAIGHT / LEFT / RIGHT JOIN.

http://man.cat-v.org/plan_9/1/join

some one-liners that are often useful:

    cat a b | sort | uniq > c      # c = a union b
    cat a b | sort | uniq -d > c   # c = a intersect b
    cat a b b | sort | uniq -u > c # c = set difference a - b

  Sed and awk have become a lost art. They're the only
  languages I know that frequently beat perl in semantic
  density. If you haven't learned them yet, you'll be
  impressed with what they can do for you.

I recently learned that sed had labels, and that you could repurpose its inner spaces as state. I sat down for a minute, while running a ~tetris written in sed.

Nice article. I actually worked for a Japanese company that uses relational shell scripting for developing web systems. They use space separated tables and have developed a set of commands to process text files. Those commands are pretty useful and easy to use. Few examples would be:

- self 1 2 fileName - selects first and second column from the file. There is corresponding tagself command which assumes the first row of the file is the column name. - Join commands which are essentially implementations of sql joins. - Sum commands. One that I remember is sm2.

   $ cat data
     a 15 asdf
     a 30 aaa
     b 13 bbb

   $ sm 1 2 data
     a 45
     b 13

So you would chain these with unix native commands such as awk, sed, grep and do some interesting stuff. Here is a very simple example shows how it would look like:

cat data | self 1 2 | sm 1 2 | grep some_expression > new-data

There is a lot other of these commands that do various sort of thing with text and html files. Anyone who understands japanese might wanna check this out http://www.youtube.com/watch?v=1MYKp7dswQg.

These tools have been bacon saving on more than one occasion. When it's time to marry your shell hacking output to Oracle I recommend checking out external tables.

Another relevant command originating (if I guess correctly) in AT&T UNIX and present in GNU coreutils: paste. There are also ways to put the whole SQL abstraction over text files, one of which is Perl DBD::CSV.