Unix doesn’t follow the Unix philosophy
johndcook.com
johndcook.com
I don't think this is the right way to look at it. Passing in additional flags is the preferred option not because users can't comprehend multiple tools, but because there's no discovery mechanism for finding out that using ls and grep together can achieve a particular outcome. The user's primary reference, the man page, provides no guidance here. Googling might help, as might asking a colleague, but by this point the cost of finding the answer begins to exceed the benefit.
What Unix misses right now is a system to help users figure out how to combine operations. Yes, it's obvious once you know how, but for beginners it's really not obvious at all. The point is that there's nothing inherently difficult about doing things the Unix way, it's mostly a problem of documentation and discoverability. There's nothing that can compete with the single man page listing all of the options for a single tool in one place.
This problem exists at higher levels too. Often web API documentation fails because it only tells you how to use the API in question, when a tiny bit of extra documentation on how to mash that API up with other popular APIs would be much more beneficial. If you want people to adopt your tool, you really need to tell them how to make it work with their other tools.
I guess this is another example of the same principle behind the success of PHP - being 'right' is nowhere near as important as having documentation that tells people how to solve relevant problems.
The standard Plan 9 shell, Rc, is described in mere ~500 lines of manpage, while Bash takes whooping ~5400 lines.
Oh, and there is no `dll hell' in P9 :-)
An example I remember from the UNIX to Plan 9 command translation page[1]:
UNIX Plan 9
head sed 10q
head -n 5 sed 5q
Although, to my eyes, the most plan9-y way of implementing a home directory would be to have '/home' as your home directory no matter what, and have it default to a bind from /usr/$username.
Email (smtp+ssl, pop3+ssl, imap+ssl, greylisting + spam and virus scanning, virtual accounts authenticating against some central source like ldap) central authentication (windows domain, kerberos domain, something along those lines that can authenticate windows clients) file servers (again, windows clients, using central authentication) web servers (just simple static file serving and proxying dynamic content to app servers) database server (no postgresql for plan 9 still) app servers (pretty much no languages support plan 9, so our custom web apps are a no go right?)
As far as I know, I can't do any of those things with plan 9. Am I mistaken? Is there something I am not thinking of that I could use it for? Do people only use it in their own private little plan 9 only networks? It doesn't make much sense to offer plan 9 as an alternative to unix-like systems if it can't do 99.9% of the things unix-like systems are used for.
Plan 9 is a research os that was made to explore the rigorous implementation of a few key ideas. You're missing the point if your criticism of it is that it doesn't run your favorite database.
If you like the ideas that Plan 9 implements you should apply them to you favorite big and nasty, full-of-compromises, gets-real-work-done OS.
For example, a key idea in Plan 9 is that all resources, from files and networks, to a processes runtime environment, to compute clusters should be represented as an entry in a single hierarchical namespace that is accessed with file semantics(read(),write()).
This is a really cool approach. It makes interacting with resources really easy. You can use any tools that manipulate files. It allows easy remote sharing and manipulation.
Want to use a printer on a different machine? Mount the printer and go. Want to write to a remote screen? Mount the framebuffer and go. Want to administer a running daemon? Mount the directory that represents it and starting messing around.
You can also create your own resources in this namespace. An example was a computing cluster with a file interface. Want to run a program on the cluster? Just mount the file an write your binary to it.
It is a neat approach and I would recommend you check it out.
Edit: Also, Plan 9 has HTTP servers, CIFS support, central authentication (factotum), etc. I suggest you Google them.
I know plan 9 has factotum, but "google it" sure doesn't help me find a way to use it to authenticate windows clients. Or authenticate anything other than plan 9 for that matter. Aquarela doesn't even claim to work, and it is actually much worse in practice than it claims to be. And it can't be part of a domain can it?
I'm not trying to be critical here, but when I ask "what can I actually do with plan 9?" I am genuinely interested in a serious answer. Stuff like "well there's an unsupported kinda buggy thing that doesn't actually do what you need anyways" is just a less honest way of saying "you can't use it for anything".
You seem to seem to be looking for "an alternative to Unix." If that is the case, Plan 9 probably isn't what you're looking for. It is, however, useful in ways that Unix is not. It probably won't replace your specific Unix needs, though it largely replaces mine. Hope that clears that up.
Since plan 9 largely replaces your unixy needs, could you share with me what those needs are, so I can try to get a better idea of what I might be able to use it for. I am not looking for an alternative to unix, I am looking for any practical way I can put plan 9 to use, rather than just tinkering with it on an old spare machine for fun.
UNIX grew to the complexity it has because it was meant to be used. Plan 9 is not used because it was not allowed to acquire useful constructs.
This parting ways with "the UNIX way" was necessary and complete in order for the system to become what it is today. Some things could be improved and made more proper and UNIXy, but for the most part it's an incredibly useful and functional system. The OP article is spot-on with this realization.
In other words, the ideal UNIX (for example, Plan 9) can't compete in the real world, because the concepts behind it (the UNIX way) don't actually compete in the real world. It's based on a set of very interesting, very elegant, and mostly false precepts.
If you "know about it", then you would also now that you cannot use it for something like the things you mention. I cannot see how your questions are compatible with "knowing about it".
Even its developers doesn't use it now --although they have ported their favorite userland tools from it to other systems.
I did not interpret your post as a criticism. Why do you think I did?
> I would like to know how I can actually use it for something
You can use it as a source of ideas for implementing operating system and network services. That may not be useful for you, but everyone is not like you. Others have found it useful to use directly(coraid).
No, ls(1) is flexible because the philosophy is to do one thing and do it well.
ls(1) lists information about files. That's what `man ls` says it will do, and whichever flags you pass, that's what it's going to do. Providing a lot of control over how exactly it does that is (imho) doing it well.
I agree that a certain flexibility is useful in the Unix tools, but a well-founded argument can be made that flexibility costs you simplicity.
This is not to criticize ls though - I would imagine that no matter how small the operation (well aside of operations executed directly by the cpu), there's still the possibility to break it up into smaller parts.
At some point convenience just trumps modularity.
ls -lt is just more convenient than a (hypotetical) ls -t | sort | xargs fileinfo | display-fileinfo
On a related note: Windows PowerShell comes closer to this ideal by being able to send arbitrary objects between pipes. This still has the side-effect though that single commands become much more awkward to use as they don't produce output that's usable on its own.
I would imagine that no matter how small the operation
(well aside of operations executed directly by the cpu),
there's still the possibility to break it up into smaller
parts.
How? sh sucks for anything involved, but I can't think of a way to simplify pipelines much.
A pair of shell variables honored by commands that let you specify output wrappers or separators would have been nice too. Think "-Q" in gnu ls, but universally supported and user-controlled.
How do you sort a multi-column output by 3rd column?. You have to write some scripts for that.
What sort option puts 1.0K, 1.0M, and 1.0G in the right order?
Someone pointed out that ls is really doing three things - listing files, sorting them and displaying them. That seems fairly accurate, and indeed it's convenient that it does.
But it's not 'doing one thing'. I'm not saying that's necessarily bad (as I just said, it's convenient), but it's not hardline Unix-way purity.
For practical reasons like those you raise, it makes sense for ls to handle its own sorting rather than leave it to the sort command, just as it handles its own formatting rather than let the user manage it. But those features _are_ something other than just reporting file listings, whatever the motivation for the feature creep. This convenient flexibility makes ls less single-task, less Unix-adherent but more generally useful on its own.
Both of which wouldn't be the job of a "do one thing and do it well" `ls`. That one would display dates as a UNIX timestamp and sizes as bytes. Aside from that, surely if ls can format information it's possible to reverse that formatting.
Here's another thought I had about the old school unix way. Old school unix didn't have symlinks. The purpose of the filesystem is to store files, right? If you want aliases, then you should keep them in a file like /etc/symlinks. Or ~/.symlinks, then every user can have their own aliases. That's the real unix way, right? Let's not take the minimalism to absurd extremes.
Do you mean that in order to truly do well its one thing (list information about files), that ls had to incorporate things like these?
-o List in long format, but omit the group id.
-T When used with the -l (lowercase letter ``ell'') option, display complete
time information for the file, including month, day, hour, minute, second,
and year.
-x The same as -C, except that the multi-column output is produced with entries
sorted across, rather than down, the columns.
Or this bunch:
-p Write a slash (`/') after each filename if that file is a directory.
-r Reverse the order of the sort to get reverse lexicographical order or the oldest entries first (or largest files last, if combined with sort by size
-S Sort files by size
* -S (and -r) are redundant with `sort` (ls -lSr -> ls -l | sort -k5; ls -lS -> ls -l | sort -rk5)
-r is also very useful (I use -tr all the time, which I can't immediately see how I'd replace with sort), and while I could replace ls -lS with piping into sort, you seem to be assuming ls will always be outputting one file per line, which is far from its most useful output mode when running it in a terminal.
No, I am assuming a "do one thing" `ls` would be used in long format (or as inode numbers) whenever it need to be worked with, and would be massaged into form by further commands. For `t`, `r`, `S` and a bunch of others you'd ls, sort and then cut/awk/whatever to format into a "short form".
Not that it matters a bit.
Also, I didn't claim these weren't useful, I claimed they were redundant. Which they are.
The long (-l) format already provides the month, day, hour and minutes. -T adds only seconds and the year. No utility can possibly provide every format option that someone might find useful. If you really needed that format, and ls were leaner, you would need to write a script (in whatever language).
> And -x is a personal preference.
Basic utilities shouldn't include so many options that are variants for personal preferences.
> None of those seem especially useless.
I don't think that they are outrageously useless, but let's set the bar higher. The core utilities should be, in my opinion, much much leaner than they are.
It is not like 'ls' is predicting weather and telling the fortune.
It lists files and directories. It has a zillion options to control its behaviour and output. I see nothing wrong with those options if they will be of use to somebody. I use 4-5 options myself and know them by heart.
I do admit that a new user can get overwhelmed if he looks at the man page.
New users don't need to be overwhelmed by the man page. -a and -l should be enough for >99% of use cases.
ls shows up in shell scripts too, but that's almost always a mistake -- or a dependency on one of the harder-to-duplicate formatting options.
Mentally, I'm translating this to "it's okay to have messy design if your software is used". This is probably the most common excuse for writing bad software.
Some things in Unix really should follow Unix philosophy more closely.
When is it okay to get messy? When it's a net win for the user experience.
Nearly every time I hear "but it's okay to be messy, because it's real life" or "I must do this bad thing to please the users" the person saying it advocated something that is an insignificant short-term gain for a tiny subset of users at the expense of very significant long-term losses for all the users.
Internal design usually does not affect end users directly, but it has tremendous indirect effects in terms of performance and reliability, which people do care about.
LS is a good example of this. Having some obscure flag might have solved a particular problem in the past, but now everyone has to deal with extra documentation, complexity and bugs.
ls output can be sorted in all sorts of ways with just a few characters of typing, even though it would be cleaner to make people use sort.
grep has a flag for counting lines that match, even though wc is supposed to be used for counting lines.
rm can remove directories with the right flags, even though rmdir is supposed to be the tool for removing directories.
I think those were all net wins for the user, both in the short term and the long term.
Yep, and that's not necessarily a bad thing. Human language is exactly the same way. Irregular verbs, contractions, and slang are all diagonal shortcuts.
There may be a few unneeded adornments to some Unix utilities, I agree. But remember that the `ls -l` that you might have typed in 1980 on a minicomputer running BSD still has the same output as the `ls -l` from a PC running Linux in 2012. That's just awesome backcompat right there.
The programming api should be as simple as possible, but no simpler, everything should be well structured and in long form and with typed arguments. e.g. 'ls' would be 'listDirectory' and it would take arguments as named parameters. It would also be directly bindable into programming languages (rather than messing around with strings to 'execute'.
The interactive shell would be a program built on top of the programming api (an api which does everything, not posix), and would contain all the abbreviations and implicit type conversions. This would probably create more choice/fragmentation/innovation than currently exists it would be more practical to deprecate commands and arguments.
1. Write libraries that do one thing and do it well. 2. Write libraries to work together.
The third rule is correct, because that is what programs essentially are: black boxes to modify input streams to output streams. I think this correction should be followed in the future.
I love the all the three points the author mentions, because together they form a a very powerful platform to work with and solve problems.
>>Why does ls have dozens of tangled options? Because users, even Unix users, are not overly fond of the first two points of the Unix philosophy. They don’t want to chain little programs together. They’d rather do more with the tool at hand than put it down to pick up new tools. They do appreciate the ideal of single-purpose tools that work well together, but not too strictly.
I don't agree with this. I can tell you that I just love Unix the way it is now. People feel they are messy because,as I said they always keep looking at them as commands and not as small programs that can be used together to solve problems. Most people don't realize this, because they don't know how to use it. It takes a lot of time to mature on the Unix platform. You have to learn what the commands do, then you need to go out and learn how to use them in combinations. Then you need to start learning the Unix way of representing data and transforming data to solve problems.
Like always most people don't realize the fact that data is a actually 'Text'. And not learning how to work with Text/Strings/Data generally creates all sorts of other problems. Although projects which are RDBMS heavy don't encounter them usually. But a lot of projects that are not DB heavy really pass through frustrating patches of time trying to figure out easy ways of representing, transforming and working with Data(Text). The most common format which is heavily abused are XML's. Followed by JSON these days. But in my experience if only they had learned how to use a little grep, cat, head, tail, sed, awk and learned to represent their data properly they would save themselves writing several hundreds to thousand lines of code. This problem is especially prevalent in the Java world. They are plagued with writing endless programs that actually do a subset of what sed or awk do. Its sad, but most people express shock and surprise while shown how their 500 line Java program is actually a one line sed command. Or how a simple awk command actually does what they spent a whole day writing. Not understanding Unix forces you just re implement Unix utilities badly!
Unix utilities were precisely invented to solve these kind of problems. In fact if you find yourself solving a problem a particular way, then ideally you should use a tool that can be programmed that particular way to solve the problem.
A tool like sed is implicitly designed to open files, iterate line by line, provide you with matching, manipulating, counting, printing abilities. It also contains variables to hold data, and do all other kinds of manipulations. If you were to keep writing programs in permutations of logic to perform each of these facilities specifically as needed(Which is pretty common, if your data is in files), you might as well just go ahead and learn sed. Because that will ensure you not just save on writing code, but will also save you time.
Now this is just sed, Awk can go further. It can help you do a lot more stuff. It gives you conditionals, more sharpened regexes, better ways of handling field separated data, variables, functions and much more. Combined with stuff like wc, less, cat, tr, xargs, grep, column etc you really get a full fledged programming language(Without even realizing that you have one :)) in which you can hack real fast without worrying about bugs.
The thing is 'Text' is actually data. And knowing how to play with Text and learning how to manipulate it well helps you solve problems elegantly, quickly and in a far more easier way than writing a specific program for each task.
As a final leap, you mature as a Unix programmer when you learn Perl. Its the King of scripting languages. You master Perl, you begin to truly appreciate the beauty of what it means to be a backend Unix programmer. Its a world whose beauty and power is known to those who program in it. It completes the total Unix equation and suddenly you recieve your enlightenement and realize why despite Unix's messiness it still is the best thing in the world.
Worse is Indeed, Better!
The argument the OP is making is that Unix has fallen into doing things The DEC Way, with gigantic monolithic programs that not only perform their obvious function, but have been turned into one of those twenty-pound Swiss Army Knives that look pretty but no-one actually uses, because they're too heavy to lug around. Does ls really need to emphasize its output with color? Better yet, in a default scheme that's often near-unreadable in an xterm that's running with its default colors? (I thank CentOS4/RHEL4 for this stunning example) No, it doesn't. Looking at ls output in a rainbow of colors is just as opaque to someone who hasn't read the man page (and grokked the system config file(!) for ls) as are the old single-character flags (with the exception of '/') that the F option gave back in '81 on BSD. Unix won over VMS, LISPMs, TOPS-20, CMS, etc., and all of those people have migrated to our platform, resulting in the world we have today.
Turns out there is a huge mismatch between real world needs and text books demands of elegance. Tools that seem to be massively monolithic are extremely helpful in solving problems and do them far too well to do them away.
Every time I hear why some programming language is better than Perl, I always see some Utopian dreams being displayed as the only thing that every matters to a programmer and nothing else. In reality there are many competing demands a programmer has. And providing them all generally leads to 'Swiss Army Knife' kind of designs. Because you are trying to provide many things in a package(That is the demand and the requirement at the first place). That sort of design might not be text book elegant, but it is extremely helpful in the real world.
There fore the 'beauty' and 'utility' design goals don't often match. This is the unfortunate problem every programmer to face sooner or later. A programmer has to abide by time, budget and some metric of quality every time.
Alas many of those demands don't lead to goals which may require an elegant design.
In early days of Unix, Lisp programmers used to call the problems Perl programmers solve, itself as stupid. Telling the user why his needs are invalid isn't a nice way to sell things.
Personally, I find sed so arcane, that if awk or perl weren't available, I'd probably actually use C to do anything more complicated than simple replacement.
summarized as "ls does not follow the unix philosophy", it points out that an example of a tool that doesn't follow the philosophy as close as it could or should. there are exceptions to every rule, what a substantive exposition.
ORANGE ALERT, send out the bubble!
What did you expect? RMS was a MIT Hacker. Which means that his background was using Lisp, ITS, and Lisp machines.