No, unlike e.g. civil engineering you're never doing the same thing twice and you're always doing something you've never done before because if you have, you could just use that code.
You can't copy-paste a bridge but you can re-use software.
You can't copy-paste a bridge but you can re-use software.
So it is extremely unlikely that civil engineering will be done twice for an exactly identical project. But no civil engineer is going to make his own table for bolt tensile or sheer strength just to scratch an itch.
Then we come to software, in software the same problems appear over and over again and we have in the 100's of possible solutions for some problems that are all almost (but not quite, of course due to sloppy design and lack of standardization) interchangeable.
Think web frameworks. They are themselves an attempt to abstract away some elements that are common, but in the meantime there are almost as many webframeworks as there are things that they were trying to abstract away to begin with (and all of them fail on one or more of the details). Or programming languages, another area in which we have re-invented the wheel so many times that the number of programming languages (1000+) will at some point probably exceed the number of human languages (about 6500 in active use, but about 2000 of those have < 1000 speakers).
I don't, so maybe it's not the best example.
But take houses for example, it's not common to see a dozen or more identical houses being built. After you've done 11, number 12 is not going to be much different.
In fact, the house sitting on top of the foundation will likely be the same project if they really are identical, just like copies of a piece of software, in other words, from an engineering perspective the job as far as the house is concerned was done with #1 if you want more than one house.
It's all about the tools, processes, materials, soil knowledge and so on that are being employed in order to solve the problem, not about what is actually built.
In the software equivalent there would be a new build process with associated terminology and tool obsoletion for almost every engineering project. Imagine workers coming to the job one morning to find that the tools they used the day before have now en-masse been declared obsolete and all their technology is slightly different, incompatible and un-tested. Then, a few weeks later at another job site it would be the same story all over. In the meantime, the engineers would be re-inventing engineering from first principles for every 5th job or so.
In other professions that would be called madness.
Software is a tool. What you do with it is an outcome.
A bulldozer is a tool. The house is the outcome.
So, we already invented a bulldozer (some piece of software) and it's trivial to duplicate it and reuse it all over the world. But when we need a new type of machine it's harder because, well, it's new.
In the software engineering world the goals is generally not to facsimile something like e.g. a house. It's to create a legitimately new tool. And since there are no real capital investments required (unlike building new tools in the physical world), and folks often value their free time at a weirdly low marginal rate, you end up with tool proliferation in a way not seen in the physical world, which makes it easy to think that we're building houses when in fact we're inventing the bulldozer.
More likely: to create a minor variation on an existing tool. There have been relatively few instances of legitimate new tool creation but many 100's (and probably 1000's) of instances of re-creating slight variations on the same tools.
For instance: build tools, programming languages, libraries (95% duplicate functionality with some other library, and of course an inconsistent interface) and so on.
> And since there are no real capital investments required (unlike building new tools in the physical world)
Software requires enormous amounts of capital, in part due to all this re-invention. We even have a name for it: NIH, and we have a term for what happens to a software project that is a few years old (technical debt).
Our tools and our processes are ill equipped to deal with the challenge and what is supposed to be building houses more often than not ends up with people re-inventing the bulldozer.
Now there are times when that is the right thing to do, but most of the times it isn't.
They're different end products, but they are not different tools.
Conceivably they could have been built with the same toolchain, but instead they were all built with totally different tools (Perl, Lisp/Python, Arc) respectively.
I mean, if those engineers were better at what they do, they'd use just one material, right?
And they have hundred or thousands of kinds of screws! Why do they keep reinventing them? They're just meant to hold things together -- one job!
And those engineers with their 'hundreds or even thousands kinds of screws', they very much favor standardization, lots of work goes into attempting to create families of compatible tools and consumables and typically a change like that will result in decades of stability in the fastener industry, it is quite rare for something revolutionary to happen, most of the changes are incremental and logical.
If you attempted to come up with a completely new thread or screw head there would have to be a reason better than 'I don't like the other screws' if you expect to gain any acceptance of you shiny new tech.
I'm trying to imagine a world where every other week the whole of engineering would be up-ended, everybody would have to totally re-train and we'd discard everything we learned process wise over the life-time of the industry.
That would be the rough equivalent of what we do in software. Ok, maybe not two weeks, but we might actually get there, life-cycles are getting ever shorter.
You're just being extremely unfair in comparing a field with ~10,000 years of development with one that has like... 100.
Everything you've called out as unreasonable is pretty much what every field, ever, has done when it first became a discipline, and your specific criticisms border on absurd.
Programming languages, for instance, are like materials: of course there's thousands, most of them are meant for research purposes, and the hundred or so used ones each represent different tradeoffs in base construction strategy. No different than the dozens of kinda of wood and concrete used by civil engineers.
Similarly, engineering of civil projects has some notable massive overruns on budget and complete design failures in even recent history. Their projects are bespoke and often use novel ideas that don't work out in practice.
The main "difference" is you're comparing a high assurance subset of one side to the general of the other, which is naturally quite unfair.
Want to compare the fields in full? I bet I can find a dozen bad handyman repair jobs for every bad JS framework or library.
And the handyman and the engineer have very little in common.
Hell, look at Asbestos, reinforced concrete, or the big dig and the construction industry is clearly less competent than generally assumed,
Software is developing in to an engineering profession at an astonishingly fast pace. It's just currently at the point where it's differentiating between tradesmen and engineers, and without that clear distinction, it's hard to compare apples-to-apples between fields.
> And the handyman and the engineer have very little in common.
Why not? You're lampooning software development for the fact that a lot of not-quite-professionals develop mixed results because they churn a lot of product onto the market. The damage done to house integrity all over the world by questionable repairmen estimating the engineering impact of a change they make is comparable.
My experience with high-assurance software is that it's similarly well constructed and engineered to high-assurance civil engineering, eg, major bridges. Failures happen, but are sporadic rather than regular.
Where did you experience high-assurance software and is there a report on how it was made? There's very few here that even know what that means although I've been working to change that over the past year or so. Always collecting more from that tiny field for any lessons learned that could be shared.
We didn't necessarily develop a lot of process in-house, because our senior engineers had backgrounds with Boeing and/or NASA, which both had extensive ideas about how to design reliable systems.
If I were summarizing, there's only really two points that cover about 90% of what you need to do for high assurance software:
1. Realize almost all bugs are because of politics and economics, not technological or engineering faults per se. That is, we make choices about how we set up our culture and corporate system which incentivizes people to create and hide bugs, while also failing to incentivize others to help fix that bug. The first step in combating bugs must be to change the fundamental incentives which create them. In particular, a focus on the success or failure of a team as a whole. Development is a communal activity, and the entire team either succeeds or fails as a unit. Someone else committed a bug (and it got merged all the way to deployment!) that brought down the system? It's because you didn't provide the necessary support, teamwork, and help with engineering your peer needed to succeed. What can you do to help them succeed next time? After all, everyone is human and makes mistakes. What's important is that the people are interlocked in layers, where one person can catch another's mistake (without punishing the person who made a mistake, because that just incentivizes them to hide them!) and help fix it before the code reaches client machines. Successes may be individual, but failure is always the system's fault, never an individuals.
2. Almost all technical bugs, in any field, are because of leaky abstractions and implicit assumptions. From abstract ones like mathematics to physical ones like carpentry. Be explicit. About every possible detail. And if you think you're being too explicit, you probably forgot 80% of the cases. Ever see the average house blueprint? Puts software engineering design to shame, easily. If you want to build something that runs as reliably as a highly tuned, expertly designed engine, you can't start with anything less than as detailed of a specification as they use. Be explicit.
Once you get to the point where you're working in genuine engineering teams rather than as individual engineers on a team and you have explicit, detailed specifications, the technologies to actually convert that reliably in to software that runs stably are pretty straight-forward.
The reason we don't see this all the time is simply that it's expensive: the politics require a lot of redundancy of time spent (eg, code has to be read several times by several people); the explicitness requires a lot more upfront planning and effort in documentation, which requires more time invested per unit of actual coding; etc.
Of course, much like we have building codes for houses and larger structures, I think it's perfectly fair to expect minimal standards from software engineers. (Especially now that, eg, IoT botnets are DoSing things.)
Of course, Cleanroom Software Engineering that does a lot of what you said without as much formal stuff as Praxis's method. Thing you should remember in these conversations is to point out to other party that what you recommend and what these methods did knocks out tons of debugging and maintenance costs. Since both phases cost labor, with maintenance fixes multiples of development cost, there were many cases where Cleanroom projects actually cost less since they knocked out huge issues early on. Can't bank on that being normal with it often being 20-30% or so more. It was around the same or less cost in many case studies, though, due to knocking out problems earlier in development.
Btw, I found a nice write-up on Cleanroom without tons of math or formality with case studies on college students if you're interested in references like that:
http://infohost.nmt.edu/~al/cseet-paper.html
I like dropping them on people in regular development in these discussions when they say you can't engineer software, it takes ridiculous skill, or it would cost what NASA's team spends (rolls eyes). Eye opening experience for some.
I was being a little facetious. Of course the technical work is highly complex -- some of the brightest minds of our time work on DARPA related projects on foundational mathematics related to automated theorem proving. (Looking at you HoTT crew and related projects!) Why is DARPA funding automated theorem proving? Because they want to create high assurance software for government infrastructure to counter the threat of AI-based cyberwarfare, and our current mathematical techniques aren't up to muster.
However, we've made tremendous progress on that problem in the span of around 100 years. By contrast, the problem of how to not incentivize your workers to do a shitty job, causing you problems later on... has been with us, I think it's safe to say, for thousands of years. (And it too, has attracted some of the brightest minds over the years.)
So in a relative way, the technical aspects are "straight-forward" to address, compared to the underlying political problem. And the more technical methods only add some assurance that you've correctly implemented the spec as defined, not that you're doing the right thing. It's certainly good to, eg, check that you're using total functions or not misassigning memory, but it's not magic. So while they catch a lot of dangerous problems, they can also lead to false confidence about the existence of other classes of problems.
> they say you can't engineer software, it takes ridiculous skill, or it would cost what NASA's team spends
It mostly just requires that we consistently act with discipline and professionalism, which are both tiring compared to not doing them, so by and large, people just don't bother. I know I don't when I can get away with not doing it (even if I know that's a bad habit).
This is one reason software is eating the world. It's actually possible to build new tools to increase productivity and start using them in a very short period of time. Sure, most of the tools are crap or marginal improvements at best. But with so many new ones being developed constantly, occasionally one offering real benefits shows up and every developer can benefit almost immediately.
I also think you vastly overstate the amount of "NIH" and vastly understate the massive amount of actual reuse.
How many developers use Apple's UIKit to build apps? Yes, a small number reject some out of the box components, but I'm sure many more developers just use what Apple provides as the building blocks of their application. Very few people write their own networking stack or HTTP library. And as much as newer programming languages get massive hype, the same languages tend to dominate the Tiobe rankings year after year (or whatever popularity metric you prefer).
In other words, focusing on what gets posted to Hacker News probably doesn't reflect the experience of the median developer.
The same can be said for nearly everything else. Insulation systems, HVAC, lighting, interlocking brick, drainage...
Choices made are made for aesthetic or engineering reasons or cost or some other constraint, rarely because the tech is 'fancy', 'new' or has been invented by the crew building your house. Bricks will be laid on plumb walls, the roof will be strong enough to support the expected load and so on and if any of it fails you will be surprised.
Note that most of this is stuff that has already been engineered elsewhere that is is combined in some novel way, and for the most part all of the materials can be used together, are standardized as much as possible and your average build crew will be able to move on to the next job without having to re-learn their whole knowledge base because what they did last week is now so '2015' that they are essentially useless unless they get with the times.
So yes, that's entirely different.
There are crappy new building materials that don't work (which you often don't figure out until years later when you have to replace your roof or windows or siding or doors), and there are bad crews that do crappy jobs.
This is really no different from software, though I'd argue it's easier for the crappy developers to continue working. This partly has to do with your website having ugly-but-working code has little impact to your typically business owner, while having bricks that are crooked and unevenly spaced -- even though they're perfectly functional -- is highly visible to everyone.
The bricklayer that does a functional-but-ugly job gets a bad reputation immediately. The website developer that does the same thing isn't found out until much later when you have to modify the code.
But in the wonderful world of e-commerce those unevenly spaced bricks are roughly the equivalent of the gap a skilled hackers needs to enter your store or to raid your db.
Also, it is probably important to make the distinction between engineers and contractors, engineers design stuff, contractors execute the designs.
In the software world we used to have these people called systems analysts, they would be roughly the equivalent of engineers, whereas the programmers were more comparable to bricklayers.
Then for a while we had 'analysts-programmers' and now the whole analyst bit has disappeared. This is a pity because I think it was a very valuable role and worthy of being an independent discipline because I believe that there were people that were good at these different aspects of the work but rarely really good at both.
https://www.techopedia.com/definition/24308/fourth-generatio...
Only in a sense in which you could do all the things a contractor does yourself, but slower. In both cases, you'd need to first learn what the compiler/contractor is doing. If anything, the compiler is a powertool that automates away the contractor completely.
I dislike comparisons of software to construction and civil engineering anyway. The two seem nothing like software. They have nowhere near enough (literally) moving parts to reflect the way software works. The closest thing to a comparable discipline that comes to my mind would be designing and building jet engines.
It's strictly due the historical development of our field. Computer systems were limited to large organizations in the early days. VMS, Unix, Windows, and less expensive hardware allowed smaller, less well-funded organizations to jump into the game.
In internal IT departments, software developers have become the "jack-of-all trades, master of none" people. We fulfill analyst, developer, operations person, and support with little to no training in any area. IMO, part of the reason software systems are so fragile nowadays is this "generalist" mentality. Mind you there are plenty of other factors that are, IMO, mostly social.
All that said, there is a tendency to think of software development like assembly line manufacturing. There was a Dyson vacuum ad that I loved. The supposed owner of the company showed the vacuum in operation and then talked about the 239 (not sure of the exact number) versions that came before it. Once the company got it right on the 240th model, mass-assembling the vacuums was quick. Building a model, letting users work with it, integrating their feedback into a new model, this embodies software development. And it's expensive along at least one of two dimensions: money or time.
(Edit: 240 to 239)
I do think software engineering as a profession should take notes from other engineering disciplines - but to compare them apples to apples is a touch unfair when our discipline is much newer, routinely attacked by bad actors (fer teh lulz, no less), and we're regularly updating our toolset to accommodate for changes in a much more rapidly-evolving landscape.
The plan:
https://www.youtube.com/watch?v=xmWmRQ1_uAE
Reality:
The decision was made to construct the bridge off to the side of the road on a separate spot, then to create a special purpose roadway across the road to the location where the bridge would be placed. And finally, to move the bridge from the construction site to the final destination in one piece.
The whole thing was conceived and executed according to plan, including a < 24 hour closure of the road and one week of interruption on the rail line (it was not possible for many reasons to have the track installed prior to moving the bridge).
That is engineering. In the 'move fast and break stuff' world that would be 'move fast and kill people'.
If this were software we'd be looking at a multi-year software project with a few hundred programmers delivered on-time, within the budget and working flawlessly on the day of delivery.
I have yet to see such a project. But as you say, 'Amazing!', the fact that you make it seem like this is 'no big deal' is exactly what is so good about it, you fully expected it to work didn't you?
The trick apparently is to make the complex and impressive stuff look so boring, I really wished that we could make software look that boring.
If bridges had scope creep the way software does, every bridge would start out 1 lane each way and be 6 lanes double decker in the middle before all lanes fly in different directions towards different cities, much less the other side of the river.
I hear what you're saying about using tried-and-true methods. But nothing else has scope creep like software does, because basically nothing else is design-only. That's why software is so crazy. It's ALL design. There's no manufacturing at any point. You can manufacture infinite copies for almost free, i.e. compile and copy and install.
If you could go from bridge design in CAD to actual working bridge in meatspace in ~30 seconds and for $0.01 then yeah, there'd be millions of horrible, horrible bridges EVERYWHERE.
The fact that the design of the bridge takes a year or two and a few million compared to the 2-5 (or 10) years and billions to actually manufacture it means that you can design, redesign, and redesign again until you get a design that'll actually work and it barely moves the needle on the total price tag.
But in software, if you redesign and it doubles the amount of time to complete the project, you just doubled the cost at least.
If it were possible to make software better, faster, cheaper by just imposing some discipline, why haven't dozens of companies done so and taken over the world?
That's a fantastic and well made point. It's akin to the cost of communications dropping over time. When moving words around the world was expensive people tended to stick to the important stuff, but now that the cost has essentially dropped to 0 we are drowning in irrelevant information.
> If it were possible to make software better, faster, cheaper by just imposing some discipline, why haven't dozens of companies done so and taken over the world?
Because the it likely is more than just 'some discipline' and because the market forces are working against you, after all nobody even expects software to be reliable so your competitor going to market with unreliable junk will eat your lunch if you slow down long enough to get it right, assuming you know what to build in the first place.
This is the heart of the problem. Nobody knows exactly what to build. Most software development is half initial coding, half bug fixing, and 90% requirements discovery.
Compiler writers actually have it pretty easy once the language is defined, which is a real honest to god spec. Comparing how long it takes (and how much it costs) to write a compiler once the spec is done would be a pretty fair comparison to how long it takes to build a bridge once the design is finalized.
And even bridges can turn into total disasters. Look at the Bay Bridge replacement in San Francisco: https://en.wikipedia.org/wiki/San_Francisco%E2%80%93Oakland_...
Or the Big Dig in Boston: https://en.wikipedia.org/wiki/Big_Dig
Compilers are typically developed in-parallel with the spec, exactly because you don't know what you want to spec before you try it out.
Optimizers have it even worse - they do not have much of a spec beyond "make it fast, quickly", so all development is trying to find interesting places in existing code.
Not by far. Actually, the woman who co-invented software engineering in their Apollo program made one for achieving similar reliability if your specs are right at around $10,000 a seat. That and Cleanroom were in production use in the 1980's making low-defect software. Many others showed up afterward with plenty of application to commercial products or significant OSS software. Here's a few.
An early one, Cleanroom, that was often as cheap as normal development due to reduced debugging:
http://infohost.nmt.edu/~al/cseet-paper.html
Margaret Hamilton of Apollo started a company later to make the one below to embody the principles they used for correctness on Apollo. The papers section is also interesting.
http://www.htius.com/Product/Product.htm
Lots of companies applied the B method for things like railway verification. Many successes that cost nowhere near what NASA spent.
http://www.methode-b.com/wp-content/uploads/sites/7/2012/08/...
Altran-Praxis does formal specs, refinement, and provably-correct (wrt specs) code with 50% premium over normal development for their high-assurance stuff.
http://www.anthonyhall.org/c_by_c_secure_system.pdf
This mentions about three, different methods being used in products or experimental projects by defense contractors:
https://www.nsa.gov/resources/everyone/digital-media-center/...
COGENT is doing seL4-style verification at a fraction of its cost with filesytem paper showing how practical it is:
https://ts.data61.csiro.au/projects/TS/cogent.pml
Some companies are straight-up using logic programming to execute precise specs of how software should work on startup budgets:
https://dtai.cs.kuleuven.be/CHR/files/Elston_SecuritEase.pdf
I can't help but note that a lot of it was spun out of NASA though.
It really does take your entire life to learn this craft. Now try selling that today.
I agree with a lot of what you're saying in this thread except that you keep missing high-assurance engineering in statements like this. Altran-Praxis regularly does what you describe minus "several hundred developers" since they try to keep the systems simple enough to not need that. Galois can do this. Cleanroom Software Engineering teams did this often on the first try. There's companies doing business requirements in Prolog with standardized components for plumbing. The DO-178B/C and similar companies are delivering all kinds of software that's tested from specs to code to whether object code really matches. Hamilton and Kesterel generate some systems straight from logical specs with correct-by-construction techniques while others in CompSci and industry do that by hand. iMatix had their DSL's and generators to do a significant chunk of this without formal methods. One company even specialized in high-availability conversions and migrations like you describe in the bridge example albeit I can't remember the name. Quite a few Ada projects also happened in defense sector with a bunch of subcontracted components that integrated painlessly due to good specs and language's features.
There are companies and groups straight-up engineering software that has few to no defects in production or maintenance activities. They actually have a significant number of customers, too. It's just that 99% of software isn't done that way. It has the problems you mention elsewhere in this thread. Let's be fair and give credit to those actually pulling it off, though. Also lends credibility to our claim more of that 99% could be as well.
EDIT Added specific links in another comment:
> It's just that 99% of software isn't done that way.
I'm aware of it. It's just that in my practice I do not run into companies that actually do this. The companies I look at typically exist between 6 months and 5 years, have a team with an average age of 25 to 30, maybe one or two older people with some in depth experience.
They will happily tell me that they write junk because they don't have time to do it right. Personally I think they don't have the time to do it wrong but what do I know.
Frustrating.
I've worked on re-working a fairly large project from a giant hairball to something a lot more solid over the course of two years and spent the larger amount of that time arguing about bad practices, the lessons learned were legion (for me at least) about why most software is crap.
If regular engineering were done this way everybody would be self-taught, would have about 30% of the picture, would not be willing to begin to take responsibility for their product and the majority of engineering projects would have fatal flaws in them.
We really can and really should do better than this if we are to take the responsibility that has been given to us collectively serious.
> I have yet to see such a project
Then you haven't worked on safety critical systems or old school embedded/firmware projects.
The alternative is 30 developers solving the same problem with software that has a bunch of critical bugs that get fixed in the first few months of release and a few hundred more minor flaws that get fixed gradually over time.
When it comes down to it the alternative is almost always preferred by the market and for good reason.
There's a reason that as-built design docs are provided. They're frequently different than the originals.
Or incorrect construction, like driving a pile in the wrong place and just adding the extra to the design.
I definitely agree. Things don't always work out as planned in any discipline of engineering. My personal experience was mechanical and electrical manufacturing, and it held true there too.
nice way to say, "the computer is doing my job for me".
which is all well and good, mind. Now if only doctors had that.
Where are the expert systems of the 70s, the ones I read about in AI books?
I think it's helpful to mention that many organizations don't even have the best of what the 1990s could offer; they are still struggling with custom integration of 5+ "system" suites, and the technical baggage of just keeping that working so people get paid and inventory management can happen. They aspire to use fancier algorithms on data they don't yet have with good quality, to provide faster/better/cheaper service to their stakeholders.
The state of the art for the agencies I serve is what I would casually describe to a HN audience as a data pipeline that often ends in an Oracle database, then a handful of web/mobile apps that should be rebuilt with Google Maps. Not rocket surgery, but they're truly in a tight spot in terms of budget to swap out systems already in play with better ones that face the future rather than the past.
Phew, I didn't know I had that much to say on the topic.
It isn't like people don't wonder why it takes so long or costs so much to, say, build a road, either. It seems like everybody massively underestimates how long it should take to do something that they don't know how to do and won't be doing. I characterize this mindset as "I need _x_ to be true, therefore _x_ must be true." It's endemic in all of the people who don't actually produce anything.
Then it's "What??? We're paying <x> dollars a seat for a configuration change????." You can't win...