The Unreasonable Effectiveness of IT

The physical world is much less flexible and forgiving than the world of software. A civil engineer can't say "this beam will break every week, so I'm just going to put a robot next to it that will attach a new been when it does". But you can do the equivalent with a process that crashes.

The greater flexibility of software implies a different set of management practices.

I think that when the Manhattan Project was started, physicists knew that it was possible, in theory, to build a nuclear bomb. It's just that no one knew how to manufacture it. Going from "we know it's possible" to "built" is more amenable to a systematic process than going from "is this even possible?" to "built". I think the article is talking about the latter. iPhones were firmly in the realm of science fiction fifteen years ago. It would have been very different to systematically try to create one. Whereas in 2007, it was apparently possible, just no one knew it yet.

Did they actually have a system, or where they throwing large numbers of incredibly bright people at it and giving them pots of money to play around with?

Management Accounting techniques are generally based on knowing what the process is, or what the output looks like, ideally both. I would suggest that the atomic and space programs have involved many problems, at various recursive levels, where the process to follow, and what success looked like, where unknown going in. Research & Development is one of the classic cases for where Management Accounting techniques are consider useless or counter-productive.

Please don't use projects such as the Apollo or Manhattan program as examples. These projects are huge and have considerable amounts of money and man-power behind them, which most IT projects nowadays don't have.

I did a quick check on Wikipedia and the Apollo program cost in 2005 dollars $170 Billion dollars.

I believe the problem with IT is that vendors have sold management that their solutions will solve all their problems with little effort or man-power. The do more with less mantra has been oversold.

The Manhattan project also came to my mind as a counter example and I was disappointed it wasn't addressed in the book.

It could be the exception that proves the rule. Internally you might be able to say it was heavily research driven. And the global resources and brain power attached to the project were unprecedented (https://en.wikipedia.org/wiki/Manhattan_Project). Plus there was the motivation of a truly existential level threat. Even given all that it still could have failed.

For some reason I didn't think of Apollo in the same way. Maybe I'm missing the boat on that one.

So a long series of blunders and mistakes results in perfection, because that was the only way that the final state could have been achieved? Regardless of how imperfect and undesirable the final state may be? I think this is a use of the word "perfect" that I'm not familiar with.

You could see a (perhaps incorrect or ultimately misleading) example of it:

1. What would have to be true for us to have reliable transmission of messages over an inherently lossy network? 1a. some kind of method to ensure that a message got to its intended destination

2. what would have to be true to ensure that a message got to its intended destination?

2a. some kind of method of ensuring that failures don't occur, or if that is impossible, some kind of method of making intermittent failures not prevent the system from eventually functioning.

3. what would need to be true to make intermittent failures not prevent the system from functioning

... etc...

Slightly misleading because I'm obviously working back from a familiar example, but I think the technique is applicable in some cases. At the very least your solution is born out of trying to make the impossible just barely possible rather than seeking to build layer by layer to achieve something possible.

However, with TCP/IP you can also see how using traditional methods also work.

Perhaps one of the more practical uses was figuring out how to scale storage performance when I was at NetApp. Since processors had stopped scaling like they had been in 2001 their CTO challenged me to scale storage performance in terms of IOs per second and total available capacity, without improving the speed of the processors.

The challenge there ended up localized around a couple of really hard sub-problems. One was that Netapp's storage model was file based (rather than block based) and so part of the problem was scaling the operation of a file state machine, and the other serious sub-problem was the creating a reliable way to keeping protocol/process state that was in "flight" in order to recover from failures. Working backwards led to what I ended up calling the "three layer cake" model which split the storage stack into parts that were loosely coherent and tightly coherent.

Proliferation of frameworks is a great example.

Think of what EJB, EJB2 and Spring did to Java. Think of Struts. Instead of fixing the real problems of the language, we went in a lot of side paths that barely got us ahead of where we started and, if nothing else, slowed progress down. Java in 2002 vs Java in 2012: Not really that much better. Working in Java before was like working with a broken bone, and instead of fixing it, the frameworks were painkillers. So we still had a broken bone, and then we were addicted to Vicodin.

We have been in a similar boat for a while in Javascript land. A few specific calls make web development to be a worse experience than using Delphi to build native apps back in 1998. This problem of repetitive code + css + a dom has led to lots and lots of libraries and frameworks, moving constantly. But it feels like they are not really moving, as far as productivity is concerned. FP inspired tools, like Elm, or React, might finally get us out of this mess, but the jury is still out.

And let's not get started about the state of build systems in javascript land. Lots of things are being written, but it's really hard to figure out why. The rate of actual progress is pretty slow.

We are also in a similar boat in cloud and virtualization. Lots of little tools that are supposed to make our life easier, but most are extremely fragile, because we build the infrastructure and sell it to the world before we understand the problem. The entire ecosystem around docker and coreos is the wild west. And then there's the distributed databases. Dozens of options, less than a handful that handle a network split in a sensible way. Just read some of the posts in the Jepsen series.

Sometimes we solve a problem well, quickly, and it's solved for good. Other times it feels that all this effort from the community is the same as being stuck in the mud. In those cases, the tools don't feel like stepping stones, or foundations to build upon. It's a bunch of different people making infrastructure that fail to learn any lessons from their competitors.

If anything, what is amazing is that we manage to get systems that work in some fashion, given that we are using so many broken tools.

I'll take a stab at it, at least my perspective on it from the "enterprise application" trenches.

It's amazing how complicated many multi-project, multi-levels-of-indirection rube-goldberg-machine-like solutions are to accomplish the most trivial of tasks. The ratio of actual "doing the actual required functional work" code to various "architectural best practices" and other plumbing is I'd speculate often in the neighborhood of 1:20.

"X considered harmful" is my personal pet peeve