Comment by kahawe | Hacker News Reader

Comment by kahawe | Hacker News Reader

This simple and completely true statement never ceases to fascinate me:

> In short, as an industry, we all stink.

We really do stink and we as an industry are nowhere near a state where there are any real standards and best practices in place like e.g. when building a skyscraper or a road. Those are fairly to pretty damn complicated tasks and there is so much knowledge, experience and best practice involved and everyone in the construction industry (should..) knows them and when it is done you can have it checked and certified that it was well built and won't crumble the next day.

I don't think anything like this even remotely exists in IT. We have a lot of RFCs describing protocols and what not but nobody can really objectively certify your serious-business software as well-built or can verify whether you applied even the smallest best practices or common sense guidelines because there are so many almost religious wars being fought over completely minor advantages and dis-advantages which simply do not matter on a global scale.

And far too many really, really actually bad programmers just get away with their mess or horrible, insane code.

We need to remember that computers are young. Really, really young. The invention of the first-ever electronic digital computer is still an event within living memory.

This can be hard to remember. Especially when you yourself are young, you're used to thinking of things that are even slightly older than you as eternal. (When you reach my age, and you are beginning to have colleagues who literally weren't alive during formative periods of your early career, you get a deeper appreciation for the fact that people -- including yourself -- have these horizons.) For example, I did graduate work on lasers, and I built my work on top of a great deal of earlier work, so I tend to think of lasers as things that have been around for a long time. But the laser isn't very old. It turned fifty this year. Many of my friends are older than the laser. Many of its early pioneers are still wandering around.

Look up the early years of engineering. Read certain chapters of the excellent Structures, or Why Things Don't Fall Down, or for a more personal view check out Chapter 20 of Mark Twain's Life on the Mississippi:

http://www.classicreader.com/book/2886/21/

The early history of tall buildings is the history of towers falling down. The early history of modern bridges is the history of bridges collapsing. The early history of steam-powered transportation is the history of boiler explosions, especially in the USA, which had a reputation for quick-and-dirty mechanical hacks as far back as the early 19th century. These situations took decades to change... or longer. Sometimes an order of magnitude longer.

And, crazy ideas about the Singularity notwithstanding, technology proceeds at a human pace. Right now, we're still at the point where substantial portions of the world population don't even have access to a computer (although the smart phone promises to change that). Very few people know how to program at any level. And that is the major issue facing programmers today, as it has been since the microcomputer was invented in the 1970s: There is more value in spreading the temporary hacks around more widely and cheaply than there is in inventing more permanent and solid stuff. Until we can meet the insatiable and growing demand for poor implementations of 1970s-era computing technologies, there's really no time or money for anything else. This condition is probably temporary, but when I say "temporary" I may still be talking on the scale of decades.

Comments with supporting book suggestions. This is the kind of comment I can get behind.

I'll second the recommendation for J.E. Gordons's Structures. The first and last chapters are wistful and heavily editorialized, but everything in between is excellent.

No. Although your point about the youth of the system involved being a critical determinant to it's reliability is correct, you make an error in naming that system as "computers".

"Computers" are reliable these days. We rarely have design problems with the actual hardware (ok, sometimes we have higher than desired failure rates, but most of the time, computers these days are very reliable). They function pretty much as we expect them to. As proof, they are so reliable that when we have a bug in our programs, we no longer even think to point the finger at the hardware. Instead, we point the finger at the software (though in too many cases devs still tend to point the finger at other people's code, when they should look at their own code first - it is the youngest, and hence least reliable, as a general rule).

The problem is software. I seem to recall pg writing an essay about this - we aren't making copies of something that already exists, we're making new things, things that have never even been dreamt of before. Of course we're going to have problems. I've watched in amazement as Rails has gone from a not-particularly-reliable framework to something relatively solid and flexible in a matter of years. MacRuby has gone from a dream to one of the best ruby implementations out there over the same sort of time frame. The list could go on and on.

But getting reliability out of a system takes time. As a rough heuristic, I think you could say that you need about a year of catching and fixing bugs in a product from the time that you stop adding features for every year that the product took to develop. If you add features, the new parts won't be right until another year has gone by (although hopefully unit tests should prevent regressions in the existing code).

Of course, in a shop with sloppy coding standards, you may never get a stable product out the other end, but I think that most competent engineers these days know the techniques that will give a reliable system - verifying inputs to APIs, Unit Tests, documentation, specs.

>We rarely have design problems with the actual hardware

Hard drives, still the main data storage hardware in most computers, are utterly unreliable.

But not for the reasons you'd think -- mechanically they're incredibly reliable these days, even when violence is done to them.

What sucks now is the software running on the embedded drive controller, especially on the most recent highest-capacity drives available at any one time -- the most common bug is for the whole drive to just stop responding for several seconds.

There is still one type of hardware error causing problems -- single-bit corruption errors. They're much less likely than they used to be, and the drive will catch them with a checksum when read back, but capacities are so high that the probability of hitting one when reading back a full drive is nearly %100. This makes RAID-5 and other parity-based striping systems almost completely unviable these days -- you'll never be able to rebuild the array from n-1 drives without hitting a single-bit error. And because of the software bugs in drives, they drop out for no reason all the time forcing you to rebuild the array (which then fails because a cosmic ray caused a single-bit error).

SSDs won't save you either, they have even more classes of software bugs present in their embedded controllers.

No, you're missing my point. Consumer hard drives are unreliable because consumers would rather save a few dollars rather than pay for a reliable system. That's one end of the spectrum. The other end is, say a bank, where databases are backed up remotely in conjunction with high-reliability RAID drives to achieve a very high level of reliability.

It's not that we aren't capable of designing systems that are reliable, but rather that we don't value the reliability as much as it costs. And that's because even a crappy consumer drive is 'good enough' for most people, unreliability and all. If it's not, we put in some sort of a back-up system to mitigate the risk.

In the early days of computers, that just wasn't so. If you're old enough, you probably remember having a C64 game on a cassette that you just couldn't get to load. Or a floppy disk that became corrupted because it got heated to over 50°C and some bits got jumbled. Go back even further when only prototypes of these devices existed in labs, and the situation was even worse. Well, most software is more akin to the prototype in the lab than a product on a shelf. It hasn't been thoroughly tested, it is still under development, we don't understand the limitations of the system, etc etc etc.

> We rarely have design problems with the actual hardware

Hah. Spoken as someone that has clearly never written drivers. Take a peek through driver code some day - you'll see workarounds for bugs on different implementations of the same standard. You'll see heuristics to, say, filter out spurious monitors reported as connected by video cards. You'll see poking random values into registers with comments like /* this shouldn't fix things but it does */.

More or less, you'll see dozens of workarounds for hardware bugs. Hardware is buggy.

Actually, there's nothing 'clearly' about it. I spent three years of my life doing nothing but writing drivers for NT 4.0 and later on linux. I'm not saying that hardware these days doesn't generate errors, I'm saying that the errors they produce are understood, and can be corrected/handled in such a way that the system doesn't fall down. You just don't see incorrect values being read in from memory or from disk, because we have error correction systems in place. You don't see the value in a register in a CPU changing for no reason. You don't see uncorrected cross-talk errors on the CPUs data buses. You don't see dodgy values being read off buses due to reflection off incorrect bus terminations. These are all problems that existed at various stages in the development of computers, but these days, it just doesn't happen any more because designers know the problems exist, and engineers are taught how to avoid them at university.

I started the clock at the birth of the first computer because, obviously [1], one could not begin to solve the practical problem of developing methods for writing error-free software before one had hardware to test it on.

I'll concede that we seem to be better at controlling the error rates in hardware than we do in software. There are lots of reasons for this; the biggest, ironically, is that custom hardware is very expensive. Therefore, there is little market for highly unstable custom hardware built by high school students. The cost/benefit ratio is all out of whack. It pays, if you're going to build hardware, to take your time and do it very carefully.

---

[1] On the one hand, at this point I expect mathematicians to jump out of the woodwork and claim that their illustrious predecessors were able to theoretically predict the future existence of buggy software as early as the nineteenth century. On the other hand, I have a gut feeling that very, very few computer programmers realized how hard it was all going to be before they actually tried it. Nobody ever does, even today. Gedanken software never has bugs.

Many "real world" building projects far exceed the originally estimated cost; some of them are not very stable at all, and scale badly when put under strain that wasn't originally envisioned.

Just one link: http://en.wikipedia.org/wiki/Scottish_Parliament_Building#Ti...

Software development is not nearly perfect, but neither is engineering of "real" buildings. My farther in law is a civil engineer, and when I told him that I believed that engineering in his discipline worked much better than in the software industry, he just gave me a disbelieving look and called me naiive.

I think the use of some of these strategies mentioned are what should be best practice. Code coverage checkers and code analysis tools, most specifically.

Reply on news.ycombinator.com