Why Software Engineering Isn’t Engineering

Is it though? I live in Boston, home of the notorious* Big Dig[1]. While particularly egregious, it's far from the only large-scale civil engineering project that's gone off the rails. In fact, I'd argue that until fairly recently, many more public works projects shared the "surprise factor" of software projects. I'd recommend Caro's "The Power Broker"[2] for a fascinating history of NY-area public works (among other things - great book all around), including how much of that process was about adapting the plan to new things the builders were learning along the way ("oh, turns out that soil is completely different than we planned...")

That's not to say that there aren't particular features that make software engineering its own special snowflake - as there are meaningful differences between how civil, structural, mechanical, etc. engineers operate. But spend some time in another engineering organization and you'll find it's different, but not as different as you think it is.

(And FWIW, even civil engineers sometimes follow "agile" concepts - a company I once worked for was contracted to design a highway, and even after the construction started, engineers were "embedded" with the builders to make on-the-fly adjustments based on the environmental factors they discovered throughout the process... I wish I could find their project write-up, but it was a while ago and the company has long since been gobbled up by a bigger company).

* As a (subjective) kicker, I'd add that the Big Dig, over-time and over-budget as it was, was ultimately quite worth it... much like many software projects!

[1] http://en.wikipedia.org/wiki/Big_Dig

[2] http://www.amazon.com/The-Power-Broker-Robert-Moses/dp/03947...

Those factors are "project management" which include concepts like "scope", "budget", and "critical path".

It's arguable if those are "engineering". They certainly affect engineering. (And likewise, some engineering constraints can feed back into project planning and feasibility.) In any case, project planning is a separate area of study. Using the author's strange definition of "engineering", it means the NASA Space Shuttle is "not really engineering" because it didn't deliver to the specifications of 50 launches per year at low cost.

A more reasonable definition of engineering that most could agree on would be, "the study (or art) of balancing technical tradeoffs against real-world constraints".

BS.

We're still exploring the space, and most of the time the things we build are good enough to work - just like the first wooden planks across streams were good enough.

We also do have the capability of designing and building the software equivalent of suspension bridges across the bay, using tools and processes like Ada, Coq, formal software proofs, etc.

Most engineering projects don't live up to your high standards, either. I knew a Blacksmith who would make circular staircases, and would regularly have to work around floors which were 4-6" higher or lower than specced. Or bridges which were put into earthquake zones, but not designed to withstand earthquakes. Or highways built with inappropriate foundations that buck like a bronco with frost heaves after the first winter passes. Or cities built under the freaking water table.

Ease up here folks. Our tooling and processes are still undergoing growth, as we take in the incredible scope of what we are capable of. Rome wasn't built in a day, and the understanding of stoneworking, woodworking, and city planning which enabled the building of Rome certainly took more than the < 100 years we've had.

I'd use the original moonshot as a better example. It was hugely complex, unpredictable, late, had a large number of last-minute problems. They essentially used Kanban to solve it - hanging a drawing of the rocket on a large conference room wall, and taping notes to each place where there was a problem; reviewing each note at a 'scrum' meeting each morning.

So Engineering has been working this way for at least 50 years. There's nothing to see here folks; move along.

One thing I certainly agree with: Regular physical engineering isn't entirely free of many of the problems I mention with software engineering, but the ability to understand and plan more effectively does appear to be there. The complexity in physical engineering does seem to be a little more tangible. I know a few civil and mechanical engineers, so I'm not plucking this from my behind ;-)

In terms of the definition of engineering, and the argument that software engineering isn't "engineering", I was taught this way back at university (City Uni, London), in their Centre for Software Reliability, and it's something I've largely agreed with. However, I think it was mainly used as a warning mechanism for newbie software "engineers" like me, to let us know that what we do is significantly different from other forms of engineering, in terms of the rigour. Perhaps it still deserves the title, but I guess that's a longer discussion.

> Engineering (from Latin ingenium, meaning "cleverness" and ingeniare, meaning "to contrive, devise") is the application of scientific, economic, social, and practical knowledge in order to invent, design, build, maintain, research, and improve structures, machines, devices, systems, materials and processes.

So, software "engineering" definitely involves the use of scientific, economic, social and practical knowledge to build, maintain, research, and improve devices, systems, and processes. So that, in my lowly opinion, makes it engineering "in the strictest sense of the term".

Arguing about semantics is extremely stupid if you are not a PHD in linguistics.

Truth is, manager's and developer's choices are why most of this isn't the case. Management might push use of a problematic technology, give too little time to assure quality, and so on. Developers might care little about making things maintainable, prefer a style which impedes it, or use fad technologies with tons of unknowns. The combination led to pervasive issues in the industry the article references.

The solution is for quality-centric IT shops to differentiate by going in the opposite direction. The same for FOSS projects. Fortunately, we see a little bit of this here and there. Altran-Praxis was delivering engineered software via Correct by Construction methodology. One student's combination of Python and Cleanroom was promising. Ada and Eiffel communities are using Design-by-Contract to aid predictability/maintenance. So, we see pieces of how vanilla IT shops & FOSS can raise the bar closer to engineering. We just need courageous groups to jump and grab it.

The last O'Reilly Software Architecture Conference had a talk by Glenn Vanderburg titled "why software development is an engineering discipline":

https://www.youtube.com/watch?v=zDEpeWQHtFU

I think the author of the blogpost does a lot of the things discussed in the talk (e.g. comparing it mostly to civil engineering as opposed to all the other engineering disciplines).

Yeah... nearly stopped reading after that opening statement. There are many reasons why software engineering is different from other forms of engineering. And maybe those reasons are enough to not call it engineering. Personally, I don't think that's particularly relevant.

But wildly exceeding estimates is definitely not something that distinguishes "real" engineering from software.

Real world engineering often has the advantage of working almost exclusively with known quantities. But add a variable, like digging a tunnel through an unpredictable underground, and sometimes projects go off schedule by years.

No engineering discipline can predict the unpredictable. Some disciplines are just younger than others, and therefor have more unpredictable elements.

The big difference with software engineering is that the costs of failure are so relatively low that we prefer to go full steam ahead instead of trying to find ways to reduce uncertainty.

This makes software engineering an outlier. Everything else is just semantics.

Modelling is a key activity of engineering - it allows predicting the behaviour of a final system. Linear systems are handy because they are scalable. A bridge to hold 10 people can be scaled to hold 100, 1000, or 10,000 people (usually) quite easily. A software processing scaling from 10->10,000 will often fail in all sorts of interesting ways in the process.

Software often fails silently. In mechanical systems, there is often noises, vibrations, or yielding to give warning and insight to where problems are happening. Software often doesn't have this inherently, but can be overcome with debugging and testing.

I think a large part of this is the newness of the field. People were building bridges, houses, and carts long before Newton. But after we had the modelling tools and theory, we were able to go much further.

(I'm a mechanical engineer, who tends to do a lot of software for controls and models, and increasing web development)

To me, engineering is a way of thinking, it's all about problem solving.

According to this guy we do mostly design and specification. Of course then you could say the Bay Bridge, the Space Shuttle, the 787 and every product that requires a long design and exploration pathway is not engineering. I don't know of anyone at Google who considers dev the same as mechanical engineering. It's more of an exercise in managing complexity, mathematical optimization, social science and design theory. But we sure as heck aren't designing for the knowns required to do what this guy defines as engineering.

But so what?

We can't efficiently reason about our designs because we can't efficiently build from reusable parts that are known to work. Sure that cup holder works in every body else's car, but it makes our fuel tank explode. There is no equivalent to things like "tensile strength" that we can calculate to say, "This will be good enough for this project, but for a different project we should use something else". We just shut our eyes and pick our components based on what the most annoying person in our team wants to use.

Many organisations make the mistake of thinking of software development as if it were like building a bridge. We think that we can choose a framework and it will reduce the amount of thinking that we have to do. Then half way through the project we are madly trying to hire Rails experts because nothing works right and it has something to do with the internals of Rails (which we were trying to avoid understanding).

Putting aside the gargantuan task of gathering requirements for a second, building software sometimes has the feel of building something physical. Because software is seen as a thing that people utilise, they think that we will eventually use the same techniques that we use for building physical object/systems. The reality is that software does not follow laws in the same way that a bridge or a car must follow physical laws. You make up your own laws in software and the only thing that is important is internal consistency.

It's a bit like creating a universe for your bridge to live in. The success of your bridge depends on whether or not you chose sane rules that other people could understand. If we were to think of it as engineering, we would need "engineers" who studied every software system so that they could understand the "laws of physics" (which explains why Rails developers get paid so much!).

BTW, I'm picking on Rails for no good reason. You can insert whatever framework/library/language/system you currently hate because it really doesn't matter all that much. Each one encapsulates it's own universe and requires us to study it to understand how it works. Our ability to extrapolate from one system to another is dependent upon whether or not the developers actually chose to imitate each other or not.

On the other hand, we have some advantages over engineering. Our universe is made up. If we choose we can limit the rules to only things that we understand very well. We also get to see the source code (unless you work with proprietary systems, in which case you have my undying pity). We don't have to discover the laws of physics by experimentation (hmm... it doesn't stop some programmers, though...). The laws might change from day to day, but we can even write so-called "tests" to alert us when some idiot has inadvertently changed the gravitational constant and caused the universe to implode.

Personally, I think the differences between programming and engineering are big enough that we lose a lot by hoping that it will become engineering. For some reason there seems to be a desire to call programmers "engineers". I hope this trend reverses and we embrace a new discipline that is more suited for our needs.

While bridges can fail after 50 years, the general state of many software applications (particularly in enterprise software) is seemingly less good.

I primary issue is lack of standardization and rigor in the field that allows for quantifying allowable defects, test coverage requirements, and understanding of failure scenarios. Simply put, so much is played very fast and loose. Fragile systems are sometimes built in haste on fragile foundations.

In say, electrical and building codes for houses (not engineering so much, but applicable), there are established standards for protection of consumers and standardization of work. In software, usually these don't exist - or they exist in small legal areas like PCI or HIPAA, and don't really explain how the software is built either, but only some of the properties. Not only do building codes exist, but the similar codes and standards exist for the parts the technicians install.

Rather than codify the practices of the craft that firm things up, everybody's still trying to figure out what those practices are. And maybe that's ok. Bridge building has been done for thousands of years, and software has been done for far less than a century.

We are still figuring a lot out. Yet, at the same time, I don't think we remember much from history, software that has "nailed it", and analyze what works and doesn't.

Software is also kind of not engineering because it's an expressive medium to some, kind of an art, hence the application of "craftsmanship" frequently applied. We are sometimes inventors, sometimes engineers, sometimes sometimes carpenters, sometimes plumbers.

These are all valid fields, but I do want for greater engineering rigor over the long haul. I think it would make things less stressful. But right now, we (as the people in the industry) are doing all these breadth first forays to figure out how software is built - sometimes getting stuck in local minima and maxima until we can upset an ideology enough to try something new - and that's partly why it feels like it does. We have a lot of people with different experience levels and different specializations, and often conflicting opinions, where sometimes none are clearly right or wrong.

On the other end, business also needs to change. A bridge is never "sold quickly", it is sold and then takes as long as it takes. Electronics can be pitched heavily, but the cost of field replacement is so remarkably large it must be gotten right the first time. However, software can quickly be replaced on the fly. More so a problem if you are working on a .com than on firmware, the need for rigor is reduced and engineering deliverable is more controllable by the desires of the business side.

Ultimately, it's still a remarkably new industry, and it itself is able to evolve quickly, as we are not realing dealing with the rules of chemistry, but rather conceptual ideas and logic. Logic and ideas are fuzzy complicated beasts.