Hacker News Reader

> in some cases, this philosophy has resulted in the "tangled mess held together by duct tape" kind of systems architecture that no one dares to touch for fear of breaking things.

Pretty much, just as Booch's philosophy has given us tangled messes of class hierarchies with cross references pointed every which way. And just as The Gang of Four gave us AbstractedTangleMessFactorySingleton.

The distinction with the Unix-style pipe mess is that it exists and works an order of magnitude faster than other examples of bad code.

Additionally, it's not a religion. Every programmer should be exposed to Unix style programming just like every programmer should be exposed to functional programming. 100% purity in either case is just creating unnecessary problems.

Amen. Extremism in programming, as in religion, is evil.

I would like to point out that most extremism is from the perspective of the listener, not the speaker. If I just wrote a book or article called "Functional programming is great," there'd be plenty of people replying with "Oh, but what about MY favorite style? It's not that great! You're just being extreme!"

When the reality is that you can't take an opinion as the whole thought. But people do. Almost always. Regardless of the subject, people will think you're a fanatic for your position, no matter how temporary or experimental it is, unless you pay your dues in apologies and waffle terms.

> I would like to point out that most extremism is from the perspective of the listener, not the speaker.

Believe it or not, there is a significant contingent of crazy, death-threat-making jerks fixated on "The Unix Philosophy". As in, an experimental Unixy project[1] that made the rounds (including on HN) some years back and got a fair bit of attention. The project had a vision with scale similar to that of Atom, which is soaring on the front page today. But unlike Atom, this project was quietly dropped from public view because the author received a full blast of nutball hate from misguided Defenders of the Faith/Purity/Whatever.

I've been using, studying, and working with *nix systems basically forever (longer than almost everyone reading this). But that experience made me rather allergic to ever using or hearing "The Unix Philosophy". I realized that it's heavily overused as an argument-stopper: "that's not the unix way (so <EOF> off)." I've since witnessed similar language used in many projects, taken in whole context, as a dogmatic excuse rather than an actual source of architectural wisdom.

[1] Apologies for being deliberately vague. It's not my place to risk riling up the hate monsters vs. the creator of that project again. I'm still absolutely disgusted that Gamergate-level human insanity was leveled at someone putting forth what was, IMO, one of the most interesting Unix tooling experiments in the past decade.

I think in terms of declarations, steps, procedures, methods, objects and a clear, logical program flow.

Which is good if you're programming in an object-oriented imperative language that has a defined program flow. But while that style is the most popular, applying the same concepts to anything that isn't imperative OO will result in unfathomable suffering.

When in Rome, code as the Romans do.

> And just as The Gang of Four gave us AbstractedTangleMessFactorySingleton.

Hey, not funny. I actually googled that pattern just to make sure it was a joke.

The Gang of Four book recommends "composition over extension" which fits neatly into the UNIX philosophy of building things by composed of smaller independent programs.

The main issue that most of the common programming languages in use notably Java and C++ have very poor support for composition thus forcing developers to have to extend things.

That's true as far as it goes, but you can write beautiful code in most languages too, even Java. The point I was making is just that "able to write bad code" is a poor metric for a philosophy. And also that "bad unix-style pipe hackery with script glue" has the advantage of actually working more often than bad code produced by other disciplines.

Can you explain how exactly Java and C++ have poor support for composition? Classes can have member variables in both languages. What else is there?

Example: How in Java do you add a method to the String class so that all instances of String gain the method without extending String and introducing a new subtype ?

It cannot be done. As an example see Categories in Objective-C.

Java's support for composition has improved somewhat with default methods on interfaces in Java 8 but it's still poor compared to the above languages. This leads to the "abundance of classes" anti-pattern you see with Java projects as you are forced to extend to introduce new functionality.

Objective-C is years along of Java extending existing classes.

Absolutely, but that doesn't mean we shouldn't try to make some progress.

I think we have actually made progress with HTTP and all the REST. But I feel that the great failure of distributed object systems in the 1990s and 2000s has left a great void when it comes to new thinking about more fine grained sort of interfaces between programs.

Functional programming has influenced so many things but hasn't really arrived in the systems corner of the world yet. Also, looking at the sort of query capabilities of online streaming databases and complex event processing systems I get the feeling that there may be a lot of potential in combining that with Unix pipes and text streams.

Publish-subscribe ala zeromq is also something that could be a core part of modern Unix.

Sorry about the half-baked mess I'm dumping on you here :)

REST is a kludge but it's easy. SOAP as much as I dislike giving Microsoft credit was brilliant, but it was hard.

How is REST a kludge and SOAP not?

> And just as The Gang of Four gave us AbstractedTangleMessFactorySingleton.

The Gang of Four didn't give us that. An army of OOP novices who were overwhelmed by all of the new choices and only skimmed Design Patterns gave us that.

If you actually read the book (which surprisingly few people have, given the number of people who have strong opinions about it), you'll see the Gang of Four are actually quite clear on the limitations or and ways to abuse the patterns.

They did, though, kind of.

The GUI-oriented 90's vibe, meaning zero examples that anybody would ever bother learning. The bizarre stuff nobody ever uses (Flyweight? Bridge?). Singleton -- 'nuff said.

You're left with what, maybe three usable patterns? Factory, which favors opaque, magical return-type polymorphism. Visitor, which is a solution in search of a problem, and also gets confused with trivial tree-walking operations. Interface, which rocks, and became part of Java, but really just shows how much inheritance sucks.

The resulting confusion is the proud sponsor of an AbstractStrategyFactoryBuilderDelegate near you.

> The GUI-oriented 90's vibe, meaning zero examples that anybody would ever bother learning.

It was written in the 90's. GUIs were what application developers did back then. Web apps didn't exist yet.

> The bizarre stuff nobody ever uses (Flyweight? Bridge?).

If you've used enums in Java or instanced rendering in a game, you're effectively using Flyweight.

> Singleton -- 'nuff said.

They caution really hard against overuse of Singleton in the book, but all of those C programmers being dragged into OOP didn't know where else to stuff all their global state so they went to town on it.

> Factory, which favors opaque, magical return-type polymorphism.

If your language doesn't have first-class classes, Factory is really handy. If it does have first-class classes, well that means you basically have the Factory pattern in your language. :)

> Visitor, which is a solution in search of a problem

The day you write a compiler in an OOP language is the day you realize how unbelievably, amazingly, incredibly useful Visitor is.

There's no good idea you can't turn into a bad idea if you just do it hard enough.

Recently I've been exploring the idea that small programs should consist of two parts; a "backend" that you can communicate with via a strict message protocol like protobuf. It can handle multiple requests because it is a mini server, the message format is locked to a "type" and you communicate with it via RPC.

The second part is a "frontend", a command-line client that does all the unixy stuff with text streams etc; but is just another RPC client to the backend.

This provides flexibility; for quick duct-tape situations you use the front end and pipe anf ilter to your hearts content. Then when stuff needs to get serious, you can bypass the command-line front end and connect directly over RPC using a strict message format which has the ability to handle a couple (doesn't have to many) of concurrent requests.

Of course this is early doors yet, but I think it might have legs in the flexibility in the text streams/hard message format debate, for very little additional work.

Incidentally, this is what neovim does.[1]

[1]: http://neovim.io/doc/user/msgpack_rpc.html#msgpack-rpc

Thanks for the solid example. I especially like the use of the NVIM_LISTEN_ADDRESS environment variable for service discovery of the backend!

They are working on making the servers aware of each other, so calling the `serverlist()` can be used to connect to other servers.

Surely a lot of that could be done with a commandline switch?

    tape_robot start -tapeid=1

as the quick hacky version, and:

    tape_robot --protobuf 'msg:start;tapeid:1'

(replace the 'single:quote;string:thing' with your protobuf message.), or

    tape_robot --protobuf_file filename

The filename could be a fifo or socket, of course. Then use a general purpose server which runs the programs:

    proto_serve tape_robot 127.0.0.1:8080

or whatever.

This (to me) is closer to the unix idea - one server program, one actual processing app. Why should my tape_robot program actually have a server embedded in it?

If in the future, I want to add authentication, I only have to add it (once) to the proto_serve program, rather than to every single application that is a 'server', for instance.

It would also allow a version which 'pre-forked' the processes, and left them waiting for the data on the socket/filehandle, or whatever.

You could do a bunch of this already using nc or similar, I suspect.

Interesting. This seems very non-unix-y to me. The `tape_robot` doesn't "do one thing", it does many things, including parsing protobuf from shell strings and files, which seems error prone and way outside its core competency.

Maybe I misunderstood @jalfresi's idea - as I understood it, it was that each command would do not only parse protobuf and unix style flags, but also contain a RPC server of some sort.

My reinterpretation was to say how about factoring out the server part, and leave the command only understanding either flags or protobuf commands - which could be delivered to the command either as an arg, or as a file given to it by an arg.

You could go a stage further by having all commands only accept protobuf (or similar), and distribute a spec/human-mapping to go with it. Then your shell would parse the args that you give to the command using the spec, and actually call the command using protobuf.

This would allow very awesome shell completion / highlighting / etc. It should also allow much simpler end-point/commands, as they'd not have to do hardly any type checking / re-parsing, as it would arrive in protobuf.

I actually kind of like that idea. Hmm...

This is actually a step toward how PowerShell (is it still called that?) works, where you pipe objects (ie. a schema) around instead of arbitrary strings.

You're sort of describing the design idea common in unix-y areas to build libfoo that does all the work, then a foo frontend that makes tasks useable from the command line. I don't know how widely used this model is, but it's certainly out there.

The difference is that in what I've described, the ABI is the common interface and an RPC server would be just another consumer of the library.

Yup, even better!

I like this idea! It seems very Unix-philosophy-y (sorry) itself; break up the programme even further, from "do one thing" to "figure out what to do" + "do it".

"Unix-philosophical"?

A lot of people don't know this but that's pretty much how Windows actually works underneath everything (DCOM, RPC, pipes etc)

That's how a lot of well-designed software works, including Windows (in general) and old-school Mac Classic with AppleEvents (not sure if OS X still does that.)

Yeah it kinda reminds me of the "plumbing and porcelain" approach of some programs like git. It also is nice that a lot of modern init process can fire up the back ends on demand and kill them after a while.

> Recently I've been exploring the idea that small programs should consist of two parts; a "backend" that you can communicate with via a strict message protocol like protobuf. It can handle multiple requests because it is a mini server, the message format is locked to a "type" and you communicate with it via RPC.

Well there was an implementation of something like that, see https://en.wikipedia.org/wiki/DCE/RPC

or DCOM, https://en.wikipedia.org/wiki/Distributed_Component_Object_M...

then there was dcop, dbus, and a bunch of others and now nobody remembers what the goal was

For a while, Linux mostly had the philosophy that your software should come with a text interface, optionally a GUI, and a C library. That library part is for solving the messes created by tangled text interfaces.

But nowadays lots of things only come for the GUI.

A lot of those are actually just task specific business logic style applications in scripting languages (often python, sometimes perl, etc).

In this case the 'library' is often the base programming language it's self, as the shipped parts are entirely glue and configuration.

Anyone can make a mess of anything if they are not sufficiently careful. This is not unique to nix.

Personally, I enjoy the nix philosophy of using small programs that do one thing well. Also, I prefer text to proprietary binary formats that _I_ can't do anything with.

As a *nix and Windows developer, I've never understood the mentality of wanting to build huge monolithic applications. But, I do think the tide is turning. I think more and more developers are allowing themselves to be influenced by ideas outside of what they are used to and that's a great thing.

>On the other hand, creating systems from programms written by different people requires more flexibile interfaces that can deal with versioning, backward and forward compatibility, etc, something that is extremely difficult to do across programming languages without heaping on massive complexity (CORBA, WS-deathstar, ...)

Personally, I think the LLVM project shows what the Right Thing is here: package most of the real functionality into libraries while putting the user interfaces into runnable programs. Everyone who wants to communicate with the functionality can just link the library.

And if we want to do it statefully, well, handling global state across a large system is basically the unsolved problem of programming.

The big problem I've found is serialization of sum types. thrift (or protobuf or any number of similar systems) is very good at serializing most of the data one tends to work with, with declarations that are suitably strict but easy to write. But it doesn't have a good way to represent "this field is an A or a B".

Thrift has unions, so does Apache Avro. Afaik Protocol Buffers is the only one where you have to string together one yourself with a required tag and a bunch of optionals.

Protobufs has unions now too (they're called "oneofs"), as of version 2.6.0 released last year:

https://github.com/google/protobuf/releases/tag/v2.6.0

Protobuf has somewhat recently added the 'oneof' feature which tries to deal with these union types.

The one downside I found when trying to use it was that the 3rd-party C# Protobuf libraries didn't yet support it.

See: https://developers.google.com/protocol-buffers/docs/proto#on...

Unix philosophy is a good place to start but needs a refresh in light of a couple of decades worth of experience.

It's always going to be a good place to start because it embodies a realistic, documented, proven and actionable compromise between engineering purity (the "formal guarantees" you describe) and just shipping something.

For a collection of related thoughts, not limited to traditional unix philosophy but still unix philosophy-centric, check out my ansified fortune clone @ https://github.com/globalcitizen/taoup

It's your viewpoint that makes you think it becomes a mess of duct tape and not some failing of the philosophy. To improve the philosophy you look outside the box not inside it i.e. not at http. Something like, "everything is a file" is an improved philosophy (but I'm cheating of course because I know this is a plan9 philosophy to improve upon unix).

I think maybe the Unix philosophy applies to programs that users (read: programmers) use in their own time, and to make other programs.

Are you saying you disagree with the use of subroutines, functions and interfaces? After all, they are essentially small programs that do one thing and do it well.

Note the part about text streams in fauigerzigerk's comment. If all functions were limited to accepting a text stream and returning a text stream, the overall program would quickly become a ball of mud.

No, I don't disagree with the use of functions or interfaces at all, but functions in particular are a programming language construct and as such not easily extended across programming languages, processes and networks, mostly due to different type systems and error states.

I think it's not an accident or historical vestige that Unix philosophy uses the word "program". It has technical as well as social implications that have barely changed in the past 50 years.

Functions and programs are certainly not an either-or kind of thing but making them one and the same would require a very different kind of operating system than we have ever seen. Say, emacs OS :)

Functions in particular are a programming language construct and as such not easily extended across programming languages, processes and networks, mostly due to different type systems and error states [...] it's not an accident or historical vestige that Unix philosophy uses the word "program". It has technical as well as social implications that have barely changed in the past 50 years.

Insightful.

Then I think you are confused or inexperienced because Unix does not treat everything as a text stream and you say functions in a programming language are different so they are better is not a wise statement. It appears you don't understand the fundamental concept and think Unix hasn't improved over the years.