Comment by Joeri | Hacker News Reader

Comment by Joeri | Hacker News Reader

Interesting to think that complexity can be a result of trying to do proper software architecture. It occurs to me that the way software is typically built probably approaches the problem of complexity in software architecture from the wrong end. We try to build software systems that minimize complexity, have predictable behavior and operate under controlled conditions. Most software has extremely low tolerances for aberrant behavior.

Looking at biological systems though, that is the opposite of how reliable systems are constructed. If we design from the assumption of failure and runaway complexity, with watchdog and recovery processes, and using emergent behavior from small discrete components to obtain large-scale results, then we can build software components that are much better able to recover from unexpected interaction and produce more reliable systems. It seems to me that this is what the theory should be. I just have no idea how to do that in the real world when building a web app, aside from recycling server processes every now and then.

You might be interested in Erlang and Elixir, where systems are designed to "simply fail" and then recover rather than trying to catch every possible error condition. These languages are also built upon the idea of using lots* of lightweight processes, which also provides a really nice model (the actor model) for concurrency and parallelism. It's worth noting that the designer of the actor model claimed that it was informed by how physical systems operate, which is akin to your biology comparison. Erlang and Elixir take that actor model and make it operate in a highly reliable fashion.

Here's a quick intro to Elixir (it takes about an hour), where you'll build a distributed app and have it automatically recover from a failure in a process: http://howistart.org/posts/elixir/1

* I had 400,000 processes running on a modest VM without maxing out.

> If we design from the assumption of failure and runaway complexity

This is a good pattern (one of many), but it is no silver bullet, either. If you overdo this, you end up with overly defensive programming, where all code is littered with catching theoretical error conditions that will never occur. That hides the actual functionality and makes code really hard to read and to change.

The solution pattern for this is to move checks at the boundary of the module, so that the code inside can make safe assumptions on the data it receives, so it can concentrate on the real functionality.

As the article says, whatever you do, you can overdo. Software architecture is always about balancing and judging conflicting appoaches and goals.

One thing that is challenging in managing the boundaries is that often times our languages don't prescribe enough layering. We usually have modules, classes / functions and methods to define our boudnaries. The real world dictates we deal with networks, protocols and content types, yet our languages, environments and communities prefer to recommend simplicity at the language level.

I think functional languages have a more healthy appreciation for layering by nature of requiring functions to act on data rather than having mutable data types. I believe if we want to reduce complexity we HAVE to create more layered systems to scope the bounds of our complexity. The problem is many people see this as over-architecting things. With time, a design can evolve such that 10 or 20 layers, as compared to the 3 in MVC we often see, could be manageable with help from our languages and environments (IDE / text editor / etc.).

I'm sorry to bring the term since some say it's a buzzword, but I wish for an anti-fragile software design theory. Ward Cunningham did demo ad-hoc parsers that build grammar through data. Nothing is known in advance. Maybe software that can feed itself back from unknown condition would be possible ?

Reply on news.ycombinator.com