Hacker News Reader

> "Do one thing well" is more than just a UNIX philosophy. It is an essential truth.

Yes, thanks. My eye detects light and passes the signal on to my brain. It does it quite well. My liver removes toxins from my blood. It does it quite well. The big ball o mud made by BigCo is a hammer that reads the ambient temperature, weighs my nails, tells me the time, and predicts the outcome of futures markets. Is it any wonder it doesn't work worth a damn?

Oh please. Your eyes are a system rife with bugs. You have a blind spot that exists for absolutely no good reason - the bundle of nerves can just as well connect to the back of the retina rather than the front. About 50% of people have a defective lens, which can only be corrected by putting yet another lens in front of it (in frames, directly on the eyeball or by ablating the eyeball in order to turn it into a lens). You can be near-sighted AND far-sighted at the same time. Your eyeballs contain imperfections called floaters that your body has no way of removing, so your brain learns to filter them out. You constantly see your own nose. Some people's eyes are misaligned and they have no depth perception. The colour violet doesn't exist, your red cones are actually activating in response to your blue cones detecting high-frequency blue light. And I don't mean they activate in response to the light, physically the red cones don't detect any of the wavelengths of violet light - they're just wired up in such a way that a low activation of the blue cones triggers activation of the red cones. Some people have a defect that makes their red cones activate in response to wavelengths too close to green light, so they can't tell the two apart. Some are just completely colour-blind. Oh, and the image you get in your eyes is rotated 180 degrees.

Your eyes most definitely don't do just one thing and they most certainly don't do it all that well. But they do an important job reasonably well, which is what strive for in any large software system.

You're right. It's only one of the most complex and intricate pieces of machinery in the universe, enabling billions of organisms around the planet to build a realistic, 3-dimensional picture of the world around them by picking up hundreds of millions of bits of information from the fastest moving particles known to man.

It's about on par with Microsoft Word.

It's the intercommunication with the web that becomes so painful. Users have no way to connect a stream of web applications, and the web disincentivized solving the corporate distribution problem.

How so? The web is already standardized on REST as the 'universal' API.

Publicly accessible microservices are becoming more and more common over time and some of them are being used to create integrations.

Have you heard of Zapier? https://zapier.com/how-it-works/

Slack bots also interact with other services: https://slack.com/apps/category/At0EFT67V3-new-noteworthy

There's also the Huggin project: https://github.com/cantino/huginn

I am thinking more at the internal-software level.

Consider an enterprise that has a simple case management app -- essentially, a help desk app for one portion of the business.

There is another app within the system that was developed 5 years before to help choose when to start a case. It's maintained by a different team, so there's no natural coordination.

If these were two Unix tools built like programmers would build, someone could connect the output of one, to a filter in the middle, to the input of the next. However, in the real organization, there is an individual who spends 20 hours a month to manually copy the information from app 1 to app 2 because the work to create a proper hypermedia interface.

The combination of a corporate-aware Huginn (single sign on, etc) and a revolution in corporate apps that treated hypermedia as a first class citizen could make it happen. It's not happening now, however.

The problem with 'enterprisey' corporate applications has more to do with the culture of the developers who build those applications.

Instead of breaking applications apart into distinct services they devs of corporate apps prefer to use OOP to create complex inter-dependent monolithic architectures. APIs for such apps are usually exposed by using a library, so a lot of emphasis is placed on access privileges at the language level (ie private/internal/public) instead of at the interface level (ie pipes/REST).

The problem isn't that the capability doesn't exist, it's that corporations are either unaware or unwilling to experiment with alternative approaches.

For example, one employee could spend half their time collecting data in Excel files from multiple members of a team, organizing it in a presentation-ready format, and emailing it up the chain.

Alternatively, the team members could each have a Google Spreadsheet where they input the data and a timed trigger fires off a bot that processes the data, saves it to a new file, and emails the link up the chain.

Both accomplish the same result. The second is a lot less prone to error/bias, and is also a lot more cost effective. But, to get an organization to adopt the latter approach requires convincing the organization to give up Excel as their canonical tool for organizing data.

I have fought that battle and it's a lost cause. Even when adequate proof of cost/time savings are presented in a business-friendly format, inertia trumps reason.

A company could theoretically hire a developer to deploy a digital assistant to streamline/automate the mundane and repetitive business processes. Except, devs are expensive, in short supply, and it's a really hard sell business leaders on the idea of adopting processes/tools that they don't fully understand.

> How many so-called "engineers" are afraid to write small, trivial programs lest they be laughed at?

Very few. Engineers love simple and reliable stuff. You might be thinking CS graduates.

Indeed. I think the problem is that "engineer" has come to have two meanings: "engineer" and "computer programmer."

The defining trait of an engineer is that he or she can't make innocent mistakes. If engineers sign off on a structure and it collapses, it's their fault; anyone who does that is civilly liable (i.e., able to be sued for damages), and, if memory serves, criminally liable as well if the mistake was negligent enough.

That standard doesn't apply to computer programmers; programmers who want to be called engineers should try thinking about what their lives would be like if they, too, could be sued or imprisoned if they messed up.

(They should also remember Paul Graham's point, at http://paulgraham.com/love.html , that prestige is the after-effect of doing something well. Done well, anything -- even jazz or novel-writing -- can become prestigious. Done badly, anything can become disgraceful, and then it starts changing its name in order to pretend to be something less dishonorable. If you don't want to be called a programmer, it's because programmers are incompetent and personally disagreeable; do what you can to change that.)

KISS has been a cherished principle for most of the past century and for good reason.

Indeed, the history of the development of very large software systems has been littered with disasters. Don't people read The Mythical Man-Month any more?

http://www.amazon.com/Mythical-Man-Month-Software-Engineerin...

Apparently not. Communication between team members has a huge cost factor ... not only ito dollar/time but also ito the quality of the final product.

Maybe within a small bubble of people who understand. I thought the idea was self-evident, but after working with people whose instinctive response to every problem is to throw code at it, I can assure you it is not.

Not all engineers. Doesn't emacs represent a different philosophy than the unix philosophy? I'd find the reference, but I have a pile of software to deal with--what a predicament.

I suspect that some engineers even make mistakes so they can heroically correct them later. I might be that engineer subconsciously.

While some small, simple programs are great, big economic contribution is mainly in large software: manufacturing control, ERP, air-traffic control, power-plant control etc.. Small software is great, but the big economic impact belongs to big software.

There is no reason why 'big software' can not be made up out of small software pieces inter-operating. In fact those are the strongest and most maintainable systems.

Small software that's made up of small pieces is still big software. In fact, most big software systems in the last twenty years or so are built that way, but that doesn't eliminate the the complexity. It makes it manageable -- not cheap.

Big software is made up of inter-operating small pieces of software: subroutines and objects ;)

Breaking something into multiple processes doesn't magically reduce complexity. You can have the same architecture within one process or spread out among 50, and in the latter case you've actually added the complexity of some IPC layer and lifetime management for all of the different processes.

That's obviously not what you meant to say. You meant: break things down into the right abstract components such as to decrease coupling and increase cohesion. You meant: do it so well that Component A can be maintained by a team that doesn't even know about the existence of Component B. And that routine changes in the requirements for one don't require changes in the other. That's really hard, especially if your codebase is a coupled mess because you wanted to SHIP IT fast. It's very hard to make a business case for big refactoring ("So we're going to spend all of this money and at the end it's going to work the same way it does now, but maybe a bit better? Can't we just spend that money fixing bugs instead?"). Also, strong boundaries between components can ironically make it harder to refactor, when it turns out that a different breakdown makes more sense. And God-forbid that other projects have taken dependencies on the components, further tying your hands.

If this was like traditional engineering, you'd have some massive upfront planning effort, dozens of committees, reviews by regulators, massive fault tolerance, etc. You'd then have a (hopefully) mini version of this same process every time requirements change or technical limitations get in the way of your original plan. Five years later you release your shiny piece of modular enterprise software. But your competitor shipped their big monolithic alternative that GetsTheJobDone in a quarter of the time and price. A good example of this is that all of the successful operating systems have monolithic kernels, even though academics and researchers favor microkernels.

So you're right, there's no reason why big software can't be made up of a bunch of small software, but there are a lot of reasons why it sometimes isn't.

> Breaking something into multiple processes doesn't magically reduce complexity.

Actually, it does.

If you break up a complex system into communicating processes then for each and every process you have a clearly defined set of inputs and outputs, memory protection, the ability to run multiple instances seamlessly without mucking around with threads and a very much reduced scope.

Decomposition into multiple communicating processes is an extremely powerful tool to reduce complexity.

Yes, it reduces complexity considerably, but nearly all big systems are already built this way and are still very complex. You cannot use this approach to eliminate complexity, i.e. make the cost linear with the system size.

Some domains where software is used to yield a tremendous economic impact are so complex that even with all complexity-management approaches at our disposal, the cost of creating them still grows super-linearly with their size.

If there are n abstract features to the system, all dependent on almost all others, you have an order of n^2 interactions. Whether you make every function its own simple communicating process, or bundle a few of them together (shifting complexity into a process) the abstract complexity of the system is still n^2.

Yes, it reduces complexity considerably, but nearly all big systems are already built this way and are still very complex.

complexity is a function of interactions. multiple processes won't do much for you if every one communicates with everybody else.

If there are n abstract features to the system, all dependent on almost all others [...].

the discussion has been about using process separation as an aid in preventing accidental complexity arising from improper coupling introduced as a shortcut so this looks a little disingenuous. my response is that if you separate every function into its own simple process, a misbehaving function need not take down the whole group.

on the improper-coupling-reduction front, process separation is apparently paramount to success, see eg. sendmail vs postfix.

Let me put it another way. Many of the software systems that make the biggest economic contributions have a very high essential complexity, and so their cost grows super-linearly with their size.

As soon as you've got a few hundred small pieces interoperating, you're not talking about something small any more.

That's true, but if you architect it well you should still be able to have an overview with every program as a functional block and inside the programs the scope should be limited enough that they are easy to understand. It's all a matter of balance. Good examples of such systems: message switches, telephone installations, routers, large scale web applications and so on. These are all very suitable to such decomposition into communicating processes.

> These are all very suitable to such decomposition into communicating processes.

That doesn't make them cheap to build; only possible.

Better expensive than not for sale.

Sure, but I think the author's point about "diseconomy" of scale still stands even when the large system is composed of small parts (and pretty much all large software systems in at least the last 20 years have been composed of small parts).

You could make the exact same claim about any bespoke piece of technology. What does it cost to make one new car? one aircraft? one piece of electronics? one moon rocket? one satellite?

It's all economies of scale that come into play as soon as you can ramp up the volume. And that's where software wins hands down, in spite of all the up-front costs.

I think that all the author was trying to say is that the cost of a single software system grows super-linearly with its size.