The UNIX Way | Hacker News Reader

The UNIX Way

asserttrue.blogspot.com

22 pointstechdog15 comments

This blog post contains mostly a small list of design principles. But those ideas are from ESR's book "The Art of UNIX Programming", where they are expanded upon in great detail, with excellent examples.

I am someone who practices the Unix philosophy on a regular basis, and cringes when I see how twisted and inelegant Windows has become (even on the programming side). I consider this book a must-read for people who know only Windows. It is also an interesting book for those who know Unix already.

The blog post attributes the list of design principles to Mike Gancarz's "Unix Philosophy" (1994) whereas ESR's book "The Art of UNIX Programming" was published in 2003.

I'm sure those ideas had been floating around in the community for a long while but was written down more formally in those books.

'Text as the universal interface' clashes quite badly with separating content from presentation. This means you're forever relying on regex's to rip data out of other applications, and relying on those other applications not to change their output.

Compare finding processes starting within the last hour on Unix and Windows:

    ps | grep 'regex representing the current amount of columns output by ps where start time is less than 2 hours'

Windows (ps is an alias for get-process):

    ps | where COLUMN <= value

The latter is much more robust. As Powershell gets more cmdlets it's going to become increasingly useful. That said, I prefer Unix for other reasons.

True, though "ps" isn't the best example, as it's one of the least portable commands I've ever seen. :) Its output is not even consistent from one Unix variant to another.

I kind of hope most Unix variants follow the /proc filesystem model of Linux. It basically cleans up these kinds of queries considerably.

But are these 'additional cmdlets' going to be available in a default O.S. install? If not, will they be automatically brought in by Windows Updater?

These rules don't seem to apply to the modern age very well. The rules "Avoid captive user interfaces" and "Make every program a filter" mean that you can't write a text editor. The rule "Store data in flat text files" means you can't write a (reasonable) MP3 player or video editing program.

"Build a prototype as soon as possible" is pretty general, but most of these rules are just good guidelines for writing command shells and compilers -- not coincidentally, two of the main concerns of early Unix developers.

But note that the overarching guideline is to structure programs in ways that are generally useful (particularly to other programmers).

I don't think the point is that there should be no bigger programs, but rather that those programs should take advantage of as many small parts as possible.

An MP3 player could have done this. For example, is there a small program that (only) extracts MP3 tags, to allow song lists to be easily constructed/sorted based on attributes such as Artist? Why should a giant GUI be the only way to perform these tasks? When one believes that the GUI is the only way, suddenly it seems "necessary" for the GUI to have its own scripting language to allow automation of these kinds of tasks. Instead, iTunes could run the "create song list based on these tags" program whenever it needs to perform that function, and use the results to update its GUI.

s/rule/guideline/ and maybe you'll be able to see the forest past the trees.

The rules "Avoid captive user interfaces" and "Make every program a filter" mean that you can't write a text editor.

Not true. Some text editors are of the UNIX nature, and some are not.

But, the fact is that you probably shouldn't be writing a text editor today. Editing text is a solved problem, and you'll almost certainly do it worse than emacs or vim if you try.

The rule "Store data in flat text files" means you can't write a (reasonable) MP3 player or video editing program.

Sure you can. Your playlists can be flat text. Your video queue list can be flat text. Your preferences can be flat text. MP3s and video can also be processed in a filter style (and often are conceptually so within the UI). Only problem is compressed data isn't in a form that can be filtered without first decompressing it. But, I regularly batch process music and audio files. If I did a lot of video work, I'm sure I'd also do the same with video.

Your program can certainly be a filter for those types of files. Sox, lame, and the ImageMagick convert utility are good examples of media programs that embrace the UNIX philosophy.

The point is that you shouldn't use a database if you don't need the things that databases provide beyond storage of blobs of data. Don't over-architect your file formats, don't obfuscate them, don't make it harder for users to use your code in ways you didn't envision. This is the UNIX philosophy...not that you aren't allowed to operate on non-text data.

Programs have been working with messy real world problems since before UNIX and the UNIX philosophy existed. The need to interact with non-text data hasn't sprung into existence in the past ten years. Before there were MP3 and video, there was typesetting, and scientific data analysis, and moon landing calculations, and neural networks, and telephone switching, and so on and so forth.

Programming in the UNIX philosophy is always possible, and, as far as I can see from my many years of software development, always wise.

but most of these rules are just good guidelines for writing command shells and compilers -- not coincidentally, two of the main concerns of early Unix developers

Every single problem of note that I can think of would be well-served by breaking it into reusable pieces. Sometimes this just means libraries, rather than small standalone programs, particularly on the desktop...but sometimes separate programs that happen to be used by a "main" program are the right way to go. Things like video and audio compression can be, and often are, performed by separate programs.

This separation is a win on many levels...on a multi-core machine, the new process can have its own resources, while you continue to work away on your media. It can be upgraded or maintained at a specific version more readily. It can be used by other programs or in a batch mode from a script or batch compression utility.

I think it takes someone who doesn't use UNIX on a regular basis (actually uses UNIX; not merely a GUI shell on top) to think these concepts are outdated and not generally applicable.

I like many of these principles, it's just hard to apply more than a few of them to a given program.

Vim and emacs have captive user interfaces. MP3 encoders like LAME write binary data files. That doesn't mean they're bad programs, but it does make the principles tricky to apply, since they seem to be useful in some cases but not others.

I'm on the Linux command line all day at work, without any sort of GUI frontend. It's got its strengths and weaknesses, just like anything else.

Vim and emacs have captive user interfaces.

With many well-placed escape routes. Again, think of them as guidelines, not hard unbreakable rules. The overarching gestalt is to make tools that can be used by other tools. vim and emacs happen to be the user of other tools (acting as a shell) most of the time, but they can also be used as tools by other programs (mutt opens your editor of choice for composing messages, for instance). They also use all flat text for their configuration and scripting.

MP3 encoders like LAME write binary data files.

So? That's their job. Their configuration is done from the command line and via flat text. They are filters.

Again, the "flat text files" rule is not that you're not allowed to work on non-text data (that would be stupid; a lot of real world data, maybe most of it, is not representable as plain text). It's that when you have the option, you choose flat text, because it is easier to work with from other programs than yet another custom database or file format. Tools using tools is the UNIX philosophy. LAME is used by hundreds or thousands of other tools. It is definitely of the UNIX nature and one of the more successful examples of it.

You're viewing things as some kind of dogma. That's not UNIX-y. Solving problems is UNIX-y, and there are some good rules of thumb that have been developed over the past 30+ years for solving problems in ways that stand the test of time. The fact that grep, sed, and so on, are used more today than 30 years ago is a testament to the strength of those ideas. Just because some folks ignore that wisdom doesn't make it invalid.

Too much of desktop linux has strayed from his #5, and it bothers me to no end (yes this means you, hal and dbus and gconf).

GConf is just text files. Open your .gconf folder.

Just xml files in a horribly tall folder hierarchy, you mean. Hand-editing xml (especially with a foreign schema) is retarded.

It amazes me that The Way of Unix is not known and understood, and most certainly applied, by everyone everywhere.

If you amend that statement to "..., and most certainly applied, by everyone where it makes sense." I would be 100% on board. Engineering a solution is all about tradeoffs and sometimes the tradeoff needs to break a long standing convention to be fast enough / have a cohesive UI / whatever. I love the UNIX way but it is not the solution for Life, the Universe, and Everything either. That is 42.

Reply on news.ycombinator.com