Why writing software is not like engineering (2008)
cs.usfca.edu
cs.usfca.edu
Software Development by contrast has no standards. You can hire two different enterprise sized companies with this and that certification in the exact same technologies, who claim to adhere to the exact same principles for best practices and what not. They mean it too, often they’ve even gone through the same sort or internal re-education processes with the same consultant companies at the same time, yet they still manage to build software projects so different that transferring ownership from one to the other, or setting up a multiple supplier open source project, is sooooo much work you wouldn’t believe it.
If computer science wants to be engineering, I need to be capable of hiring an architect who makes plans that can be implemented by any supplier, and I need to be able to switch supplier/contractor 2-3 times during the process without issues because the plans can only be implemented one way. We’re probably a lot further away from that world today than we were 30 years ago. The only company that seems to have smelled the music on this stuff is Microsoft and their cloud.
We have plenty of laws and standards regarding handling private information, payment systems, independent pen testing and so on. So it's incorrect to say we have "no standards" at all. Where it matters, we do.
But to try and standardize a process that's evolving faster than a standard can spread is pointless.
This doesn't make software engineering any less engineering. Engineering isn't about standards. It's about the design, construction and use of systems.
Compare this to when we build a new city hall. A massive project involving almost 500 smaller local contractors, in a business that actually moves forward pretty fast. Some of the materials they ended up using weren’t even invented when they began. The project was finished a head of schedule and under budget.
I know construction isn’t always like this. We’ve had our share of catastrophic projects, one elder-care center in particular, but compared to buying software it’s almost child’s play, even when things go wrong.
So I think it’s perfectly fair to say that software engineering isn’t engineering. If you can even build a simple webpage, without two suppliers disagreeing so much on the “standards” that it takes 6 months of governance to achieve compliance, then there is just so far for the industry to go before it’s anything that even resembles engineering.
A better solution would be standardize a backend apis and setup standards around how data is queried and stored. Next standardize how that data is presented to the user and how the user can manipulate that data. None of this requires a specific technology and sounds like decisions were made by people that don’t understand the problem they’re trying to solve or are being influenced by others with something to gain (like your big Drupal house).
We have actually attempted to do this on a national scale in Denmark because it was silly to have 98 municipalities handle organisational data (and other things) in 98 different ways. So we created something called rammearkitekturen (framework architecture) which specifies what an “employee” is and how you’re supposed to transfer them between systems. The suppliers hired to build this, and these are companies like IBM, Systematic and Netcompany somehow still manage to build things in such a manner that the next company needs to spend a whole year beyond their estimate to get things running after a public procurement process that saw an ownership change. Some of these are complicated, bur most of them aren’t. Or at least they aren’t supposed to be. Now, either IBM is completely devoid of talent, or maybe software “engineering” just isn’t “engineering” at all. At least not on a level that is in any way acceptable to us as enterprise customers.
I’m still not sure why we tolerate year long delays on software projects. It’s not like building a new hospital never gets delayed, but the consequences of those delays are typically bosses losing their jobs and companies paying delay fines (or whatever you call that in English), where as with software the consequences are just a few shrugs and a joke about that pesky IT.
I actually don’t disagree with you as such. My point is simply that when you hire two talented engineers from the same background to build the exact same thing. Then they will both rather easily understand what the other did and how he/she did it, unless it was software.
The experts need to continue to audit, refine, and integrate on the standards. Locking in a specific technology is shortsighted and will cost a lot later.
In construction a standard isn't "use this make of hammer". It's "achieve this effect".
We have somewhat similar standards in software development. Like, “build this type of API here, capable of handling this type of data model”.
The issue comes when I can buy that API from IBM and then can’t transfer it to Netcompany next year because they don’t agree on how to implement a REST API or how to handle the data model beyond the API. I don’t have that issue on construction. I can have a supplier hired to hand the electric installations on 300 buildings at a hospital, who go bankrupt half way through, and then I can painless hire a new contractor to finish up.
As long as software doesn’t have anything that comes near to this, it’s “just” art. I’m not saying this as a negative thing, but when two “engineers” who have gone through the same education and follow the same international “best practices” build the same thing, so differently that they can’t understand what the other build, then it’s just not worthy of the engineering description.
You can hire different people to work on code too. There's no one person per repo rule.
That's not true in the context of medical and car software. See
In medical, automotive, aerospace, military applications, the software that goes into the device is kind of considered part of the object. It needs this validation for obvious reasons, such as the long lifecycle and critical nature of these products.
But in cellphone handsets, PC software and firmware such as in IoT home devices, and even IT infrastructure like SoHo routers especially but also other devices, there is no agreed upon standards that I am aware of. Even in payment applications, there isn't much there. Large institutions and banks get by often by using "legacy" infrastructure like mainframes to keep the COBOL stuff running, as it has been vetted over decades.
Going further, the architecture of infrastructure, firmware and software should be examined more closely. Recently, IT architecture has been exploited over and over (Kaseya, SolarWinds, MTA) having quite asymmetric effects. In the past it was worms and drive-by malware from janky websites, today it is extremely costly disasters that make the news but roll on bye until the next turd is flipped over.
The problem space is very difficult to define, since "software" and "architecture" are such general terms encompassing basically anything. I rate the difficulty in analyzing these problems for sufficient solutions as very high. When I consider neural-net AI applications, it seems to get even more murky.
Is anybody aware of other software and infrastructure standards that exist or are in research which could apply to more general fields, or to corporate/business IT infrastructure, or to home devices, etc?
You don’t though. This is just an artefact of the structure of your industry, being limited as it is by physical, geographic, and capital constraints. Software has none of those (capital used to be an issue but is no longer - this is perhaps why things are more flexible than 30yrs ago as you note).
You think you need these things because you see your role as an integrator - but software engineers already perform that function at a different level, via libraries and protocols.
In my weird, crazy world, engineering is all about discipline (sound of a riding crop slapping a leather-clad thigh), and follow-through. It's a lot less about slide rules and differential equations.
I believe that the Romans that built aqueducts (often using coerced labor -otherwise known as "slaves") were engineers. They had a lousy numbering system, no concept of trig (although a few had heard of Pythagorus), and not even a slide rule; yet they were able to build massive, well-designed structures that still stand, to this day, nearly three millenia later.
I can't use some Web sites that are just over a year old, because their components went belly-up.
But I also consider software development to be a craft. That's really my approach.
When I write software, I make a personal level of investment into the outcome. The project must complete. Even when I'm working with a test harness, on a project that is nothing more than an experimental platform, I treat it as if it were a shipping product, that I plan to display in the store window. In fact, I often revisit truncated experimental efforts, in order to scrounge snippets. Since they are already "ship" quality, those snippets tend to be quite useful.
A basic habit of mine, is that all of my code "ships," even when it doesn't. That's a craftsman attitude. Getting the project to "ship" state is an engineering effort.
But that's just me. YMMV.
I think the issue is misunderstanding what it means for something to be engineered. There is a correct answer. (Especially because you can always optimize for cost or a different quality metric)
As a result, I regularly see the insides of many snotty noses. It was the main reason that I stopped looking for work in the field; but I was shipping stuff from my very first project[0]. My entire career has consisted of architecting and implementing shipping solutions.
Meh. Whatever creams your Twinkie. I'm not losing much sleep over it, but it is annoying.
[0] https://littlegreenviper.com/TF30194/TF30194-Manual-1987.pdf (Downloads a PDF)
I believe that they can both coexist. I think that the prior art is centuries old.
You have less repetitive process in software because once software is built, copying it and reusing it as-is is free. In construction if you need to build 10 twin buildings, each of them is a project on its own. In software that'd be building one piece of software and deploying it 10 times.
Furthermore I question the notion that software projects can accommodate more "fundamental midcourse design changes". Can you spend 2 years writing a dedicated Windows desktop application and then midway suddenly switch it to being an Android application? That would be fundamental. Reskinning the UI or replacing a dependency with a similar one is not fundamental design change.
Software can't sustain fundamental design changes, unless we misinterpret what is fundamental and what isn't.
Basically we need to be careful with our analogies so we don't cross from trying to extract insight into flexing our useless cynicism about how software sucks because it's not like making a car in a superficial ways.
My take on this is to allow for rapidly evolving practices in an industry where key standards may not be updated to over a decade in some cases, or even cease to exist at all due to emerging fields.
What this means is that a responsible engineer has to make a defendable decision, using experience and judgement to justify a particular course of action.
As my engineering manager used to say to me, "Would you feel comfortable saying those words again, just adding on Your Honour'at the end?"
Whenever you have abstraction, you have lost what it means to be an engineer. There is really only 1 way to engineer something as you end up optimizing for various quality metrics, most often cost.
There's only 1 correct answer for a bracket. You want it cheap and as strong as it needs to be.
How many correct answers in my code? Do we optimize for clean code? Fast code? Do we change programming languages or go with a familiar highly supported code? There are 2 similar libraries, which do we choose?
Programming is a hybrid of science, art, tradition, and authority. I think it pays better because of this complexity.
That being said, I'd trust an engineer to program something before I'd trust a programmer to build something.
That's an interesting observation. I'd say engineering does have abstractions, but they're much better. For example, treating an RSJ as a simple uniform piece of material is an abstraction. In reality it is a crazily complex mesh of crystals. But the abstraction is simple and well studied, so it can be used reliably.
Programming abstractions are less reliable. Even commonly used ones like "the filesystem" and "the heap" are vastly more complex. It is almost impossible to remember all the possible hazards when you use them.
If a civil engineer designs a bridge, and it crashes and people die due to a design flaw, he or she can be sued, and even put in jail.
On the other hand, software comes with a license that explicitly makes its authors not liable for design flaws.
When was the last time you had to agree to an end user license agreement before crossing a bridge?
It is a professional title and while it isn't usually validated, thus only those that sign projects with legal liabilities do make the final exam, every single university in the country is only allowed to have certified engineering degrees.
I am aware of similar practices in other countries as well.
"...30% of all software projects are canceled, nearly half come in over budget, 60% are considered failures by the organizations that initiated them, and 9 out of 10 come in late."
In 2021 (as well as in 2004) it is meaningless to talk about "all software projects" without specifying domain, size, scope and project etymology. "Software" is too vast. It could be an app to tell the time with Mickey Mouses hands, or it could be the control software for a joint strike fighter.
That's much better than I would've thought.
In a billion dollar construction project, everyone understands that if you want to change the design or significantly expand the scope halfway through, you're going to have a heck of a lot of increased costs & time.
In software, it's very common to add scope or major design changes at any stage without understanding that the same cost and time increases will apply.
For example saying 90% through the project:
-Software: "we need a 2-way interface added between the complex software we're building and some other complex software"
-Building: "We need an enclosed concourse built between our new building and the next building over across the street."
It is obvious in the later case that there will be a lot of extra time & cost. While in the former, higher level decision makers on a software project don't understand the comparable difficulties.
If construction companies figured out they could outsource labor to countries with far lower currency values and thereby not-pay for extra labor, they'd do the same.... oh wait...
Not in my experience, which may not generalize. Your experience and my experience together won't really give a complete picture.
What I experience is this: Even on small projects I've had people say, "okay, can you just add this one thing in the next day or two?" And they don't understand until I walk them through in detail how that one thing is nearly as complex as the initial project, so they have to choose how important it is, and if it's important do they want to wait to release the whole thing or do a release in stages.
Of course my experience is not necessarily going to represent the whole.
Also when outside consulting groups are involved, I often see predatory behavior: they often agree to do any change asked for and don't bother to explain the costs and time involved until a deadline is missed and the budget is running out. At which point they say "the changes you requested have pushed us over the deadline and the extra time is going to cost $X".
There has always been a dilemma about what to do with bright young people who want to become computer programmers, but who also want to get a college degree.
My mom taught a programming course at a satellite campus of a state university in the 80s. The course was listed in the computer science department, but it was basically Programming 101. Her students were mostly grown-ups. Her students didn't face the dilemma because they already had college degrees, or they didn't. They were getting programming jobs after one year in the course.
My mom's advice to me was that programming was too easy to justify 4 years of college level training, and that I should major in something "real" to use her words. (She has a masters in chemistry). But actually her idea was to use college as a way to build up domain knowledge that could add value on top of programming skill. She thought there would be a need to be able to differentiate myself from the likely flood of people who were taking the 1 year course and getting programming jobs.
This post would come close to trolling if it weren't for the fact that there remain ongoing debates to this day, even in HN threads, about the merit and need for college level training to become a programmer. I'm not the only person wondering about this -- it's a real and hard problem. I actually like the idea of a decent career path for someone who is not interested or cut out to succeed in the college environment, even though I thrived in that environment myself.
Much of what we do involves not just telling the computer what to do, but communicating with other programmers (through code). This is where elegance and craft can make a significant difference to the success of a project
- It takes several years for someone to prove themselves as a developer among their peers and this reputation does not usually spread outside the confines of a single company. People have to prove themselves all over again each time they change company.
- Big, high-exposure, high-influence software companies like Facebook, Google etc... force arbitrary tests on prospective developers (candidates); but these tests say nothing about a person's skill as a software developer. These tests are reliable at selecting for status-oriented individuals who can regurgitate solutions to common programming puzzles and present themselves as fast-talking and quick-thinking (I.e. Ben Shapiro archetype). These companies have a huge impact on global software development tools and philosophies; their recruitment decisions have a huge impact on deciding industry best-practices. The huge influx of superficial, fast-thinking, fast-talking software developers within the centers of power is making the software industry look like the medical industry 200 years ago when all reputable doctors insisted that blood-letting, electroshock therapy and lobotomies were reasonable treatments for most ailments. Their arguments relied on superficial, smart-sounding observations and appealed to the authority of large medical institutions. Modern developers like to leverage their associations with financially successful corporations 'sources of authority' to project themselves as top experts and to promote their agendas.
- Too many people who have no sense of curiosity at all joined this industry purely for financial reasons over the past couple of decades; these status-oriented people will end up running all the big tech companies and enforcing their ill-conceived philosophies onto others, silencing inquisitive minds. It's a numbers game; the voices of the many high-status, fast-talking, loud-mouthed idiots will drown out those of the few quiet, low-status geniuses.
We are entering the dark ages of software development.
Why writing software is not like engineering - https://news.ycombinator.com/item?id=1207721 - March 2010 (72 comments)
> Engineering is less risky than software because engineering experiences fewer constituent component interactions. While minor changes to one part of a car's structure can easily affect the crash robustness of another, it would be unusual for a design flaw in the dome light to cause intermittent engine stalls. In a home construction project, you would have to work pretty hard to make a toilet flush every time someone rang the doorbell.
While software has more "component interactions", each component is also more consistent than physical components can be. One example: using stainless steel screws on an aluminum plate can cause your building to collapse. [2] That's a bit of an extreme case, but many fields of engineering, like electrical and chemical, regularly struggle with small changes leading to wild differences in the emergent phenomena.
> The final reason that software is more difficult to get right than engineering projects concerns midcourse design changes. The physical world is not as malleable as the insubstantial world of software and, consequently, clients simply have lower expectations. Congress does not go to NASA halfway through a moonshot and ask them to go to Mars instead. Most engineering projects are able to actually use the waterfall design method: determine functional requirements, design, implement, test. For most software projects, this is a recipe for disaster.
Very few projects are built using "waterfall", and everybody I interviewed had horror stories of requirements changing mid-construction. At least a couple people I talked to had to move buildings after they were built. One ex-industrial engineer worked for Boeing and his sole job was to manage midcourse design changes.
One story I didn't put in the essays: a naval engineer worked on an oil rig. They needed to add a new piece of equipment, but it would have caused the rig to violate certain liveability constraints. It could stay on for short periods of time, but not 24/7. So they built a nearby platform to store the equipment, and then airlifted it in by helicopter every time they needed to use it, and airlifted it out immediately right after.
> Writing software is most similar to writing fiction novels.
Almost everybody I interviewed said that writing software was very similar to their old work in engineering. A lot of people I've talked to outside the project said it was most similar to cooking, or planning an event, or doing surgery, or whatever. Most fields are like other fields, which is why interdisciplinary studies teach us so much.
The way we approach problems is the same. We are not doing research or science. We are using our judgment to pick a particular solution from a large solution space to a problem(s) that must meet different user specifications/requirements.
Don't get me wrong there are massive differences but IMO they are largely cultural and macro economic related. I'll get to this in a bit but first I want to rant about what the author misses.
The author is conflating levels of design abstraction. He sees that traditional engineering has coordinated on common engineered components for example the Bethlehem Steel plant was so prolific it became the de facto standard for steel wide flange shapes for the whole country. So the author is correct in that now since engineers aren't busy designing every single minutia of every beam like they did back in the day when there were no standards they can start from a higher complexity point and focus on more complex designs. However, software developers can do the exact same thing. Software definitely has "engineered components" we call them frameworks, libraries, modules etc. They allow us to "spin" up new tools and working concept level designs very quickly since we're not busy coding all of our data structures from scratch every single time.
I'm not going to get into his risk argument since I'm already ranting too much, am lazy and it's a complex topic but I'll just say that the ability to revert a software project back a couple months means you only really lose labor/productivity and not capital. Reverting a building or part of one back a couple of months is a nightmare scenario and on average "rework" is ~25% of a typical construction project's final cost (this includes both capital and labor costs but the point still stands).
The midcourse design change argument is flat out wrong. I've been on several large projects that never got built or were unrecognizable from beginning to the final product. There are many reasons for this but traditional engineering design software is mostly to blame since there's more flexibility now (compared to the 1960s) in how people can render buildings so as a fraction of a projects lifespan projects stay in the "development" phase much longer as opposed to the "building" phase which means designers can change A LOT of the project before they need to get buy in from all construction parties and agree on what they're all building. This is one reason among many why construction productivity has gone down and costs have gone up.
Alright so what are the biggest differences between traditional engineering and software development? IMO the macro economic moment means that software is higher reward financially which means that software developers have TONS of room to grow in their field either technically, managerially, operationally, professionally etc. There's A LOT of choice and room on where you can take your software development career. This is not so on the traditional engineering side of things. There is limited growth so many many firms are top heavy and put in de facto brakes, check boxes and licensure requirements on the career ladder since there simply isn't enough room for the new incoming grad class every year. It's essentially a pyramid scheme. This makes the work culture very traditional, stagnant, and risk averse. Nobody wants to fall out of line because they can be replaced. There aren't interesting niches to explore or exploit since everything is very full and competitive.
The growth in software allows developers the breathing room to sharpen their skills and generally the culture in software really encourages this self improvement. Not so in traditional engineering. Gaining productivity in some new skill means rocking the boat and challenging orthodoxy. The margins are already slim and the boss really needs the reliable bonus he's been depending on so why take a chance on something new? Just wait your turn in line like he did.
Allowing developers the time to build technical skill means there's incentive for employers with larger margins to leverage those skills. So developers get more responsibility this is a virtuous cycle which leads to faster career growth. IMO junior devs get much more responsibility earlier in their career than their traditional engineering counterparts.
My take is on the pessimistic side but I can't in good faith encourage a smart, ambitious, technical/analytical person to go into the traditional engineering disciplines. EE/CS as far as I can tell is still the best route.
And some math formulas are considered beautiful, and art is in the eye of the beholder, so even the optimal solutions might be art to some.