E. W. Dijkstra Archive: “Why is software so expensive?” (2009) | Hacker News Reader

E. W. Dijkstra Archive: “Why is software so expensive?” (2009)

cs.utexas.edu

It's a good piece, and Dijkstra's views on this remain relevant, even if the whole thing is ultimately an exercise in trying to build an ornate analogy that will somehow survive translation to other realms. Anyone who has been in software long enough to have tried to explain it to someone who isn't in software has had to go through the same basic process. That in itself ought to be somewhat enlightening. In any event, the article is worth the read if only for this:

>> How do we convince people that in programming simplicity and clarity —in short: what mathematicians call "elegance"— are not a dispensable luxury, but a crucial matter that decides between success and failure?

Amen, brother. Great question.

I disagree. There's a realm of fields to which software engineering applies. There's fields where correctness, elegance and formal methods definitely matter; those involve complex products like airplanes (RIP, A400 victims...) or nuclear plant control systems. Or academia, which cannot progress with imprecise thinking. You can't run Facebook's operations the way you'd run a bakery's website's.

But then there are heaps more fields where correctness absolutely does not matter, where a shiny cover over an awful spaghetti ball is enough, where you can lose half the client's data and say "sorry, crash" and he'll just nod and shrug, and where this is a lot cheaper and faster than hiring a professional and doing things properly, where the determinant of success is not the engineer, but the management team, their connections, and the sheer amounts of money that investors are willing to pump into the company to win the market (money they will mostly give only to the first competent person that pitches them the idea, giving the pretty-"MVP"-in-a-week team the edge over the formals).

It's a trade-off between time and cost to market, and product quality; it won't change until the second way beats the first on the first two dimensions as well as the last.

The problem is that software usually moves from the first realm to the second realm over time.

True. Very true. But if you don't let it start in the first realm, then most of the time, there's nothing created, so there's nothing to move to the second realm.

Yes, but when this happens, the company can usually afford to hire thousands of engineers and brute force the issue, or the product is unpopular and in the process of disappearing.

Exceptions (like Whatsapp) are legendary for the concentration of wealth in the hands of a few engineers that follows success.

And what's ironic is that EWD's preferred approach actually increases the cost of (at least the first version of) many pieces of software. And his article was "Why Software Is So Expensive".

I mean, his approach would have prevented Microsoft from ever existing. Or Linux. Or Apple. Which means we never would have had the PC revolution. We would just have a software industry that produced software right, and code that didn't crash, but we'd have maybe 1/100th of the software industry that we have. We'd miss 99% of what computers have brought us. I'm not willing to make that trade-off to get software that never breaks, even though I hate it when software breaks.

I'm not sure one could describe Microsoft as "fast and dirty". Gates' BASIC interpreter is a famous example of technical ability...

But Dijkstra would oppose Gates' BASIC interpreter, because it wasn't built using Dijkstra's approach of mathematical reasoning, and therefore couldn't be trustworthy.

I mean, his approach would have prevented Microsoft from ever existing. Or Linux. Or Apple. Which means we never would have had the PC revolution.

In an alternate continuity where Dijkstra's methods prevailed, other companies and products would have arisen instead, or perhaps the same but under different circumstances. There is no reason to believe the PC revolution would have never occurred, as that was largely a result of commodity hardware (where QC is much more stringent than in software). The software would have been written anyway, but it might have been less bad.

We'd miss 99% of what computers have brought us.

Strictly speaking, no. The quantity of software might have been less, but the quality and thus the actual features would have been either the same in aggregate, or, as intended, far more progressive and in line with the academic state of the art.

It's always quite fuzzy to discuss alternate histories in this way, but for what it's worth, your conclusions quite simply don't "sound right" based on my experience of what and how software has contributed to the world at large. I don't see any way in which "the Dijkstra method" leads in a similar time span to a world in which it's quite so easy for your run-of-the-mill mom-and-pop small business to so simply spin up their own little corner of a messy world-wide network of people to whom they wish to peddle their wares.

It seems really easy to me to navel-gaze at how "bad" we think software is, and forget how great it has proven to be for so many people outside our bubble.

The fact that software engineering applies to so many fields is exactly the reason why I often find it problematic when it's treated as 'one thing' in a discussion. While the process of writing software looks deceptively uniform (files with code), the purpose of said software can be anything in reality (and it can even be used to create an entire virtual representation of (a) reality).

Perhaps 'software' is too broad a concept to discuss without qualifiers?

It's a bit like lumping together essayists, lawyers, analysts, journalists, poets, novelists, philosophers, etc. under the label 'writers', and then try to have a meaningful discussion about hiring practices, writing standards and writing tools. This doesn't really make, as the requirements of a junior reporter are quite different from those of a junior lawyer. The fact that they both write words on paper doesn't change that.

I suppose they could discuss the quality of different pens. Our equivalent to that might be code editors...

It's really not clear that the spaghetti code approach is better time-wise over anything but the immediate short term. In my experience, any advantages fall away once you get out of hackathon territory—a matter of weeks, not months.

At a point, even if absolute correctness is not important, you just spend so much less time debugging your old code and it takes so much less effort to add new features (ie less need to rewrite or extend old code to support the new) that you end up producing more overall.

The problem is that these improvements are non-local. Yes, over a month, you might get more done per day if you slow down and write less hacky code—but it won't help you get your thing out today. In the short term, it feels like you're always more productive being hacky because, day to day, you get specific things done faster… but, over the long term (or even the medium term!), you're actually less productive. The human mind is not great at accounting for time and effort.

This is exacerbated significantly by tight deadlines and constant pressure. And the sort of companies that have those problems also tend to overwork their engineers, which has exactly the same sort of non-local consequences. (If you work 80 hours this week, you will get more done than you would have. And the same is true next week, and the week after—but your base productivity is falling so, after a few months, you will be less productive in absolute terms!¹)

These local improvements are very tempting—and sometimes they're actually justified! There are definitely scripts and programs that only matter in the immediate short term, and you shouldn't worry about making them elegant. But people apply this far to widely. If your program will matter in the medium term—if you're going to be iterating on it over multiple weeks—it's probably worth maintaining a bit of discipline throughout and writing more elegant, maintainable and readable code.

¹ I remember seeing a pretty compelling study about how long work weeks affect absolute productivity, but I can't find it at the moment.

Oh, I agree with you entirely.

Let me reword/summarize my comment: there's r-strategists and K-strategists. It's usually better in most relationships to be with K-strategists. Where I disagree with Djisktra is in the idea of a world without r-strategists; not because it's not desirable, but because it's highly unlikely.

For anyone else who gets here and, like me, isn't familiar with this terminology, it appears to be ecological terminology for differentiating between species that select for quantity versus quality[0].

[0]: http://en.wikipedia.org/wiki/R/K_selection_theory

Hmm, I'm not sure we disagree. I think you're looking at a larger picture. I think the scope of most software was somewhat narrower at the time these opinions were formed, and I believe in his mind Dijkstra was thinking about the structure of individual programs more so than the architecture of systems or technology based business. This strikes a chord with me because I was thinking just other day about the fast vs. good trade-off, and I have come to the conclusion that "fast" doesn't mean poor quality. It means, or should mean, that you build fewer pieces, attempt fewer features, but the individual components that you do build are still well structured, well-designed, well-implemented, and don't close off avenues of evolution as the business scales. In other words, for me he is saying that code must always be written well, as simply and as elegantly as possible. You still retain the freedom to choose how much to write.

>> One of the problems caused by the non-understanding software manager is that he thinks that his subordinates have to produce code: they have to solve problems, and in order to do so, they have to use code. To this very day we have organizations that measure "programmer productivity" by the "number of lines of code produced per month"; this number can, indeed, be counted, but they are booking it on the wrong side of the ledger, for we should talk about "the number of lines of code spent".

Amazing how we still haven't figured this one out yet.

>> I know of one —very successful— software firm in which it is a rule of the house that for a one-year project coding is not allowed to start before the ninth month! In this organization they know that the eventual code is no more than the deposit of your understanding.

It's so hard for me to imagine that such a software firm could actually exist.

Even though I've noticed the advantages of thinking first and coding later on a personal project done in a more relaxed period, I guess it's just too hard to justify your cost without having deliverables (code commits, in this case).

> for a one-year project coding is not allowed to start before the ninth month!

And here we are, asking team leads why development cannot begin till week 2 of a 24-week project.

It is a really strange and funny industry we are in.

I suspect managers and programmers might be counting the weeks from different points in time...

I think at the level of complexity of current day products, the "think for the first 75% of the time" is flat out impossible. Certainly some of the complexity is incidental, but much of it is necessary for the sort of product we have today.

  I think at the level of complexity of current day products, the "think for
  the first 75% of the time" is flat out impossible.

Maybe, but from my experience we don’t do enough up-front thinking. Indeed, if our products are more complex these days then shouldn’t we be doing more thinking? If our products are more complex then diving straight into code is one of the worst decisions we could make. No wonder we end up with tangled balls of yarn.

> Amazing how we still haven't figured this one out yet.

Because humans tend to gain scientific knowledge much faster than they learn from their social mistakes. Managers deal with human relationships, but they really can't apply scientific knowledge to social knowledge.

> I guess it's just too hard to justify your cost without having deliverables (code commits, in this case).

Some domain might do, where the expectations or the competition are very high: military, security, console programming.

> for a one-year project coding is not allowed to start before the ninth month!

Early in his career Dijkstra had to write an operating system for a machine that had not yet been built. So he wrote the whole thing on paper, working only from the spec. When the machine arrived, his OS pretty much worked. This experience shaped his view of programming for the rest of his life. He came of age when you had to write programs where there was no machine (or you couldn't often get to the machine). That's not a typical situation now - Dijkstra was the very first programmer in Holland. (He later met a group faced with a similar delayed delivery that had not begun work -- he was amazed to hear that they were "waiting for the machine".)

This sounds like the waterfall approach taken to an extreme. (Although, even within the waterfall, having time to iterate your requirements a few times, and then your design a few times, would be a good thing...)

Why is the title labeled "2009"? I presume it's because that's when the transcription was last revised, but the actual essay was published in 1982; anything written by Dijksra cannot be more recent than 2002.

Minor correction: This was publish in 1982[1] and was transcribed from a PDF in 2009[2].

[1] https://www.cs.utexas.edu/users/EWD/ewd06xx/EWD648.PDF

[2] https://www.cs.utexas.edu/users/EWD/transcriptions/EWD06xx/E...

This was written in the late '70s, not 2009:

https://www.cs.utexas.edu/users/EWD/indexChron.html

While Dijkstra sits in his ivory tower , patiently figuring out the most elegant algorithm , in the startup world as Zuck says is all about moving fast , breaking things in order to find out whether we even need the algorithm in the first place.

When my boss comes to me with the next story , I can't tell him that , "Hey , I think I need two weeks for this as the system is becoming overly complex. I think I need some time to refactor the system into something simpler and more reliable .. "

I've seen an example of this first hand. We had an image editor that allowed users to manipulate and rotate layers using like you have in Photoshop. Instead of using straightforward affine transforms , they came up with some weird , extremely complex implementation I'm yet to get my head around .It isn't even documented as to how it works.

Finally when the time came to make it behave differently , I reimplemented only the new behaviour using affine transforms and simple trig and routed the flow through the old code or the new one using flags.

Not the most elegant solution but that's how it is in real world projects. It's only the output that matters. The internal complexity is not something the customers gives a shit about. They want their feature and they want it now. And they're not even sure what they want.

So the need for rapid iteration to discover what you need to build in the first place , and the quest for the most ideal and mathematically elegant solution will always be in conflict with each other.

Dijkstra did a lot more for moving the programming needle than Mark Zuckerberg. Sure, Mark made more money, but that's not the only yardstick by which we measure achievement (fortunately!).

Without a bunch of people doing basic research none of us would have jobs.

I have great appreciation for people like Dijkstra and Knuth and the work that they do. They write operating systems , compilers and databases.

But coming down from the ivory tower of academia , do you really need to go to the trouble of writing your custom CMS in Haskell after proving it's correctness in Coq ?

As a further example , the succesful company KissMetrics built the first version of it's product in a month and shipped using SQLite (!!) as a DB.

In their own words , they optimized 100 % for shipping speed.

Software doesn't exist in a vacuum , it plugs into a business.

The amount of rigor and research that must go into a piece of software depends on the domain and business requirements. To insist that all software be written to such a high standard is unrealistic and will never happen.

I'm yet another software engineer who (most of the time) writes CRUD apps as quickly as possible so my boss pays me on time. I have no pretensions of being a 10x or being the hero who drags the world out of the dark pits of poorly specified and mathematically inelegant software with his superior intellect.

Shipping a product with sqlite doesn't contradict with having well structured code.

Also, it's fine to write a prototype to get to market faster. But building further on top of that prototype is dangerous, because it will become harder and harder to make changes to the system and still have it behave correctly. And once the incorrectness of the system gets beyond of what the business will accept, it's very hard not to do a full rewrite, which is often very dangerous as well.

Granted, this is still what often happens ...

Part of being a good software engineer is knowing when to do the "wrong" thing (implement something poorly to ship fast). But you must be aware of the trade offs. For early stage companies/products that are struggling for a market it often makes sense to hack things, but at a company that has a large and growing customer base, it rarely does.

Well sure, but you can find your beliefs in ivory towers too. Also, argument is central to academia or my impression of it anyway.

But coming down from the ivory tower of academia , do you really need to go to the trouble of writing your custom CMS in Haskell after proving it's correctness in Coq ?

Dijkstra was nothing like this. In fact, he was probably one of the more practical academics (if that even makes sense to some people). He advocated top-down, logic-based methods for proving correctness that were based on incrementally transforming a high-level program specification into a final product fit for implementation. He likened it to composing music.

He also deeply studied paradigms that were firmly imperative in nature. Besides his work on semaphores for concurrency control, his deductive system of predicate transformer semantics and the Guarded Command Language built on top are some of the most comprehensive notations for imperative languages. He was also on the team that developed ALGOL 60, so it's quite likely you owe most languages you've used partially to him.

That and your conception of an academic using the analogy of the "Haskell CMS verified by Coq" is largely a straw man, though I can understand why you used it. HN is pretty crazy about these things, so it's easy to get caught up in the thought bubble thinking it's the norm.

May you be cursed to spend the rest of your days writing PHP.

There is also tons of examples of companies going bankrupt because they did not thought they would be consequences by shipping a half-baked product. It's just that sometimes you need to prioritize shipping and sometimes you really need a good product quality, it depends on a lot of factors and the business you are in.

It is this approach that leads to the "Full Employment Act of Security Professionals" everywhere.

> I'm yet another software engineer who (most of the time) writes CRUD apps as quickly as possible so my boss pays me on time. I have no pretensions of being a 10x or being the hero who drags the world out of the dark pits of poorly specified and mathematically inelegant software with his superior intellect.

You could stand to learn something about the Hell of fighting other people's quickly written crap just to get your job done.

Dijkstra doesn't do any work, he's been dead since 2002.

I don't think that the user you're answering to is criticising research perse but rather the irony of an academic talking about real world development scenarios.

Consider that this essay is 33 years old and remains still relevant today such that we can read it and appreciate the issues.

I wonder if Facebook will remain relevant for that long?

Given that facebook isn't an essay - it provides different value to an essay, under different circumstances, to different people - I'm not sure what you're going with that question.

Part of the domain of application programming is about "agility" because, from the viewpoint of business, if they want something changed you should be able to do it fast.

My experience is that agility is best served by building on top of a well thought-out framework. For instance, shops that use Ruby-on-Rails have answers to most of the common problems that turn up. If you're dealing with pre-framework applications in Perl, PHP, ColdFusion, Java and who knows what, the simple task of adding a field to a form involves touching code and schemas that are all over the place.

Of course your boss and customers only care about the output. But if you need to make sure your output is correct or want to make it easier to add new things, a good design is crucial

> The internal complexity is not something the customer gives a shit about. They want their feature and they want it now.

None of this is false. At some point though, the internal complexity may grow to the point such that you can no longer deliver the features they want right now. What then?

That point is dangerously close. I don't now what to tell them . Not everyone appreciates the benefits of not adding ridiculous features and non stop changes that overcomplicate the system while not adding much value. In fact many of the features aren't even used.

Also when a project is outsourced , then the outsourcing firm is only happy that there are lots of changes as that means more billable hours. They're not going to push back. And neither can the developers working there. This is yet another danger of completely outsourcing your project without any technical oversight.

Hopefully , I'll soon be working on something else . ;).

> I don't now what to tell them .

You have to manage expectations. Not just do whatever it takes to get things working, until they don't work/can't be changed anymore. It all looks the same to the people who haven't taken a look at the plumbing themselves.

Then the good developers responsible for most of the codebase leave for somewhere else using their experience as CV points, and the firm rotates through large numbers of inexperienced or incompetent engineers (who can't get a better job) as the team inexorably grows to several times its original size, whilst the number of new features dwindles until everybody is doing maintenance on the spaghetti ball, and existing features start breaking slowly.

I see you've also used BackupExec.

Wow. I've experienced exactly this process twice already, and I've only seen the inside of about six large companies. I didn't know it was such a common way of things unfolding.

Did you also have the "ticket racket"? This is when the CTO sets up some kind of ticketing system, and uses the increasing number of breakdowns (and useless "fixes" that break again with the next build) as a KPI to justify increasing budget. "We fixed 150 tickets today! My team is working overtime on this, they are really dedicated and mission-driven, their spirit and commitment is amazing." The CEO nods, suitably impressed - he really has a crack tech organization. He's getting a great return on investment on his highest budget department!

Bonus points for inserting a thick PM layer in between to "manage" the rapid expansion of tickets.

Yup. Another possibility (if a company is fortunate enough) they'll be able to snag a dedicated "maintenance developer" willing to deal with the mess for 2-3x typical salary.

Your example is a tad ironic. Why wasn't the original system implemented in the straightforward way? Unless there's a good reason, your example is a pretty effective demonstration of what Dijkstra is warning about.

Yes it is. But on the other hand, the system has worked all these years , thousands of customers used it and got their jobs done. At that time they needed quick prototypes to settle on a good design.

When it comes to translations and rotations, most of the algorithms out there use 4x4 matrices, dual quaternions, axis-angle rotations, or Euler angles. There may be some emerging work using geometric algebras.

So when you say "extremely complex" I think quaternions, because i^2 = j^2 = k^2 = ijk = -1 .

I once had the opportunity to refactor a lot of disparate implementations of 3D manipulations into a single math library, and migrate all the various algorithms to quaternion-based calculations. I actually had time to examine and understand a lot of different ways of doing the same thing and validate that they actually were all doing the same thing.

But then we all got canned when the contract was used to wipe someone's backside. I think the former customer tossed all that custom code in the trash and decided to do everything with MATLAB. I hope that's working out well for them. And by that, I mean I hope they are put into a persistent vegetative state during a prison riot.

Ha !

See. It happens everywhere.

"And by that, I mean I hope they are put into a persistent vegetative state during a prison riot"

That's a little more invested in your work than i think is healthy.

He addresses this point:

But apparently, many managers create havoc by discouraging thinking and urging their subordinates to "produce" code. Later they complain that 80 percent of their labour force is tied up with "program maintenance", and blame software technology for that sorry state of affairs, instead of themselves. So much for the poor software manager. (All this is well-known, but occasionally needs to be said again.)

    > When my boss comes to me with the next story , I can't tell him
    > that, "Hey , I think I need two weeks for this as the system is
    > becoming overly complex. I think I need some time to refactor
    > the system into something simpler and more reliable .. "

Sounds like you should blame Zuck, not Dijkstra.

What happens when the cruft becomes so overbearing that you can no longer deliver features? Not only can you no longer deliver features, but you now have to communicate that the state of the system is unworkable and you need X weeks to fix it. Thats a much more uncomfortable position to be in than say "Hey, to enable us to keep developing this system we need to allocate some time to refactoring. If we don't, there is a danger that the system becomes too internally complex too manage and starts to damage the reliability and flexibility of our product."

99% of software won't be around that long.

Sorry, but my experience shows the opposite.

I've worked in multiple jobs where some piece of software was an incredibly complex ball of mud, where things were left without refactoring because no-one currently in the company understands how they work, and no-one can spare the time to do the refactor properly. Where changing a method or property unexpectedly breaks some other, apparently unrelated feature. Needless to say, this made adding new features a pretty risky affair.

This happens all too frequently to believe in your 99% statistic. For example, it happens with some pieces of software at my current job.

It happens all to frequently because the 1% tends to be very long lived. If 99% of babies died in the first year of life, all of your friends would still be more than one year old.

The software 'industry' moved in the opposite direction.

Reply on news.ycombinator.com