Is Software Engineering Possible? | Hacker News Reader

Is Software Engineering Possible?

vanemden.wordpress.com

157 pointsdanarmak118 comments

Edsger Dijkstra famously derided software engineering as "how to program when you cannot".

But if being able to program means being able to write large programs and get them perfectly correct, then no one has ever been able to program, the great Prof. Dr. Dijkstra not excepted. We all have puny brains next to a million lines of code. That being the case, the study of "how to program when you cannot" is critical for those of us who actually need to write these large programs.

I like to say that software engineering is the art of managing complexity. Designing abstractions to be as general as possible, with interfaces as simple and clean as possible, and minimizing coupling, makes it much easier to think about how the major components of the system interact. And that's the key: if it's too hard to think about the system at a high level of abstraction, because the interfaces have too many special cases in them, more mistakes will be made.

I don't think I can agree with the OP that formal verification is the only thing that can fairly be called software engineering. I do agree that the more verification we can figure out how to do, the better. But I would point out that well-designed abstractions are still important even when you're doing formal verification; indeed, possibly more so, because without them, the proofs get to be even more complicated and difficult than they need to be. It's really the same phenomenon: systems that are hard to think about are also hard to prove things about.

Designing abstractions to be as general as possible, with interfaces as simple and clean as possible, and minimizing coupling, makes it much easier to think about how the major components of the system interact.

I think this is a false statement. In fact I think you're confusing modularity with abstraction.

Abstraction hides complexity, and abstractions are leaky; how the whole behaves can be strongly influenced by behaviours that leak from the abstractions. It took a couple of goes, but eventually we realized that DCOM and CORBA were poor ways to design distributed applications.

Abstractions should be as general as necessary, where necessity is subject to judgement.

Make code too abstract, and people may not understand it because it's not concrete enough, full of indirections and compositions and factory-factories; make it too concrete, and people may not understand it because it's too complex.

Creating an abstraction to share some code may pay off if there's sufficient clients for the code and they all have a similar usage profile, but it may be a false economy if the clients are few or the usage varied.

Modularity isn't an unalloyed good either. Too many modules and you may write code that acts optimally in a local way, but suboptimally globally because it has no access or awareness of other modules' behaviour. Modules designed around an abstraction may be substituted, but designing for module replacement is more often than not over-engineering. It's strongly encouraged by modern OO testing approaches, but IMO it puts the cart before the horse.

I'd like more people to realize that abstractions represent unknowns, and that any given implementation needs to be tested in an engineering materials sense, to find out what its force limits are, how it reacts under scale and pressure, so people using the abstraction are not just thinking about the abstraction, they are considering the real thing they're dealing with as well. It saves a lot of time going down dead ends.

That we're not all programming in machine code is evidence that some abstractions are useful.

The most vital abstraction is that most software is blithely unaware of the physical construct embodying it.

Abstraction isn't the enemy. I think much of what we call good "software engineering" is identifying, designing, and implementing the "right" abstraction.

I'm not sure I understand what you're saying, but it's also not clear to me that we really disagree.

Abstractions are often leaky, yes; sometimes unavoidably so, and then those leakages have to be taken into account in the design of the system; but when leakage is avoidable, do you not agree that it's important to avoid it?

I do agree that there's such a thing as over-abstraction, and over-engineering in general. I recall a discussion here on HN, oh, probably 2 or 3 years ago if not more, where somebody said they had written an app and then hired a programmer to maintain it, and the programmer started by "enterprisifying" it, adding lots of dependency injection and factory-factories and suchlike, rendering the code almost unrecognizable to the initial developer. Without having seen the code in both "before" and "after" states, I can't form a firm opinion about whether this was really an improvement, but I could easily believe that it was not. Certainly if I were the initial developer, I would want to know specifically what the benefit of each of these changes was expected to be.

I think Rich Hickey's "Simple Made Easy" is a perfect example. He enumerates an intellectual toolbox and why it's useful. He admits "you may not end up here [using these tools] but there's no reason not to start here." There are clearly cognitive and technical benefits to arranging software in particular architectures. That seems like engineering to me. You can't make an ideal engine either; it depends on whether you're powering a car or an airplane. Solving domain specific problems through technical architecting is about a good a definition of engineering as you can get. Software certainly fits that definition.

"But if being able to program means being able to write large programs..."

What does it mean when someone is only able (or only wants) to write small programs?

For example, small programs that can work together to form a "system" for managing something.

Is that not considered "software engineering"? If not, what is it called?

(Embedded software engineering? But not all small software is "embedded".)

What has a better chance of being error-free, a small program or a large one?

Should "ability to program" be measured by how large a program one can write?

For example, if someone writes a small UNIX utility, but does not write a large program, should we assume he does not know how to program?

As Jtsummers says, once you're talking about small programs working to form a system, they're not independent small programs anymore -- they're modules in a larger system. My point is, if you design the modules well, you don't usually have to think about the internal structure of each module when you're trying to think about the behavior of the entire system. Such design is exactly software engineering, in my view.

> Should "ability to program" be measured by how large a program one can write?

Yes, absolutely: that's a very important measure of programming ability. As I think Fred Brooks pointed out in The Mythical Man-Month, each order of magnitude in program scale requires new techniques and a new way of thinking.

I wouldn't say it's the only measure of programming ability; there is a place for finely crafted small modules. But on any project that's setting out to build a substantial piece of software, you need at least a certain number of people who know how to do that (then they can teach the rest). If nobody knows how to do it, the project is likely to fail.

> if you design the modules well, you don't usually have to think about the internal structure of each module when you're trying to think about the behavior of the entire system. Such design is exactly software engineering, in my view.

I've been trying to find the right way to phrase this, as it's more of an inkling but touches on my current academic (self-study) interests but also is something I've observed over the past decade or so as a professional developer and tester in the industry.

The engineering component of software engineering needs to study systems engineering. There's a great deal of overlap. My conjecture is that we move from "programming" to "software engineering" when we are more interested in the connections between components rather than the components themselves. Much like how systems engineers aren't doing the work of designing the new car engine, but are overseeing that work and the work for the production line itself and coordinating with the glass manufacturer for the new windows. They care about how different parts of the final system (at the level of the car, or the level of the whole assembly process) will function between each other. Systems engineering (as a discipline) has the tools to handle this sort of division of labor and abstraction and responsibility.

And it's not really that we don't care about internal behaviors. But that's not the primary concern of the overall system engineer. It's the component's engineer who's primarily responsible for that, while the system engineer needs to focus on how that portion fits into the overall architecture.

No program is small, as it must include the operating system.

Now - you could go down to machine code, or work on embedded systems, etc etc - but the point remains. It's not just the software you are writing that's part your program, it's also all the other, existing software (and physical designs) it makes use of, or interacts with.

Edit: Might be called "software systems engineering", or maybe "dev ops"?

That's not really accurate. I can write a small program that does not include the OS, but rather assumes it. Just as a vehicle designer doesn't have to include the gas or electric distribution infrastructure in their design, they can assume it.

If I'm conforming to someone else's interface (say the C standard library), that's equivalent to an electrical engineer assuming various standard interfaces for the equipment they're designing. It's a design consideration, but not a design task. So the project itself may still remain quite small.

When I'm designing the interfaces that others are using, or designing the interfaces between 100 [0] smaller programs so they can interact effectively with each other and their environment, then it's a larger engineering task.

[0] Arbitrary number.

And likewise, the OS relies on the ISA, not having to know about chip design.

Actually, it is thinking in general that is the art of managing complexity. Software engineering is no different from any other type of engineering, in that it is also the art of making reasonable compromises.

Making reasonable compromises is certainly important. But I think software engineering is different, because the state spaces of programs -- particularly large ones, but even most small ones -- are astronomically larger than those of any other kind of engineered artifact. Most procedures contain one or more conditional branches, which test some aspect of the state of the system and then take one or another action based on that. If you're not careful, you wind up with a system where you have to think about all of those conditionals at the same time, which becomes exponentially difficult as the system grows. Other kinds of engineering certainly involve complexity, but they don't involve this much complexity.

We can use mathematics to formalize what a "well-designed abstraction" is, so there is not much of a conflict between the two ways of thinking.

Please elaborate!

I can't, my understanding of category theory and universal algebra is only enough to sniff at it. Joseph Goguen's work is a good direction to look to if you are equipped with those tools. He cites Parnas's work which is the easier to grasp analysis of modular abstraction. Here is a representative paper:

A hidden agenda Joseph Goguena, Grant Malcolmb http://ac.els-cdn.com/S0304397599002753/1-s2.0-S030439759900...

Calling the discipline Software "Engineering" was the worst thing that could be done, at least given the current state of the art. Because of that, people thought you should write software the way you build a bridge or a car. The waterfall method came out of that, and it took decades to realize and recover.

Software development is more reliant on discrete math (and more generally the ability to reason abstractly) than math that helps model the physical world. Hopefully, Professional Engineering bodies will one day realize this and will adapt their program accreditation criteria accordingly.

"Because of that, people thought you should write software the way you build a bridge or a car."

The reason people think that is because they have no idea how the engineering to build a bridge or a car is done.

Note, the term is Software ENGINEERING not Software Assembly or Software Construction. The engineering work necessary to design a bridge or car is actually not TOO dissimilar to designing anything else. You have requirements (often vague, conflicting and changing) and constraints and you have to trade things off against each other to come up with a workable design solution.

The construction is just the compilation. Or perhaps the final coding/debugging. Or a combination. In construction, tiny engineering issues are often resolved by the construction crew at the site. Sometimes with input from engineers, sometimes not.

The problem is that building software merges the engineering (design/architecture) with the construction (coding/debugging/compiling/installation) that requires a rather broad range of skills. Most people working on small teams have to jump back and forth between the big picture (more engineeringy) to the small details (more constructiony) and back.

Finally, software development is no longer a new thing. People have been doing it for a half century. There are many people in the field today who's parents wrote software. And a few who's grandparents did it. It's high time we stop trying to draw analogies to building bridges or designing cars. Software Engineering is its own discipline with its own unique challenges and dynamics. Better to just take it for what it is.

No the waterfall method was a misquote by a non-tech reading a report on iterative, software development. The misappropriation took a life of it's own for control-freak managers. Anyone wanting an example of where sensible people went with software engineering can look at Cleanroom from the 80's:

http://infohost.nmt.edu/~al/cseet-paper.html

The results speak for themselves. A modern approach is Altran/Praxis Correct by Construction method getting nearly zero-defect software. Some provably correct in SPARK.

http://www.anthonyhall.org/c_by_c_secure_system.pdf

Today's work can add symbolic analysis (eg KLEE), assertion-driven test generation, languages like Haskell, certified compilers, and so on. What you're seeing in mainstream isn't software engineering. It exists, though, in s niche by academics and industry.

> the worst thing

On the contrary, the worst thing was the abandoning of the sound engineering principles and the obligatory professional certification. Since then, the trade has been open to, and flooded by, know-nothing lazy half-wits who, should they find themselves in any other industry or medicine, would be immediately disqualified (or, rather, would not be admitted in the first place).

Mainstream engineering and certification developed thanks to litigation, insurance, and regulation. These are things that we don't have in the software industry yet. UL is called "Underwriters Laboratories" for a reason.

I work in product development, and when we send a product out for certification, they don't look at functionality, but only safety and regulatory conformity. Is the grounding wire green with a yellow stripe?

All of this exist on software. Try working on software in medical, aerospace or finance.

For the sake of your health, I won't. ;-)

But it would be interesting for the rest of us to have an overview of how software is done in those fields. i'm guessing that those are also areas where litigators have refined their ability to prove fault and quantify injury in dollar terms.

> Since then, the trade has been open to, and flooded by, know-nothing lazy half-wits

In other words, web "devs" who are now calling themselves "full" "stack" and flood all walks of software development with subpar nonsense in the name of silly startup "mvp".

Even worse, "senior engineers" after taking a 6 months javascript course and working 3 years in the field.

Why do you think they are prevalent and successful?

Because the market doesn't reward useful, quality work as much as it rewards throwing crap together and investing in marketing.

But do we mean quality and usefulness as technical attributes of a program's construction at a point in time?

Or as beliefs of a customer about a product, or a company?

The skills to get to the latter are often different to the first, and both are expected in many modern software engineers. The later is much more about iteration, experimentation, empathy, and judgement. It's not worth polishing areas of a system which you anticipate abandoning based on the results of an A/B test. Writing it and abandoning some meh code can still be the best engineered choice in a product development process.

The engineering of the development process matters more than the program.

I'd argue that success in modern software engineering (at least in Silicon Valley) is at least as dependent on understanding social processes, as it is on computational ones.

Oddly enough I think there's an analogy in mainstream engineering. If you build a supermarket, it has to be engineered for safety. But the engineers are not responsible for whether the mall serves a purpose, such as selling products for money.

Because the market is still paying for software licenses instead of paying for services. It rewards writing as much code as possible to sell it. And then deprecate it and write more and more.

While I totally agree with you on the 'it's not engineering in the engineering kind of sense'-part, there's another interesting point a friend once brought up in conversation:

At least where I'm from, telling people what I do, I often get labeled as a 'computer person'. You know, kind of estranged from the real world and not good for anything practical (and mind you, after college I shed a lot of my admittedly nerdy past). Coining the term 'software engineer', with the word 'engineer' being something people can relate to and see it as a respectable profession, I think can help a bit in conveying that yes, what we do is a real and important job and no, we are not all geeks that totally like to be in their dark basements all day.

Software engineering is no less 'engineering' than, say, designing computer hardware - if anything, simply because there is often a piece of software hiding inside a microchip, but also, more generally, because the process of logical design of digital electronics is not much different from writing a program - especially if it is designed using what is essentially a programming language (Verilog, VHDL, etc.).

If you write software that needs discrete math, like probabilistic stuff or graphs, it will need discrete math. If you write software that is meant to simulate, like maybe even simulate real bridges, or video games, or generative art, non-photorealistic rendering, ... it will involve some of the math that helps model the physical world. It makes more sense to just learn a lot of things and apply the union of that to software, even ideas surrounding managing complexity in architecture (Notes on the Synthesis of Form?) and so on and expose that through oneself to "software" as a new universe where creations can exist; than to quibble about what's more important and what's not in a prescriptive way about what others want to and don't want to do. Of course, I'm doing the latter here. :) Such is the cycle.

Software crafting is very much like building a bridge back in the days before xrays were around to detect cracks in metal and before metallury and casting were very reliable - that was back in the 1800s when anything made of metal could and did fail at any time - including bridges - quite often catastrophically. A high percentage of bridges folded, over time. How do you "engineer" what you can't measure (such as whether a metal beam is fatally flawed by a large crack inside?) You don't. We're getting there slowly.

>>The waterfall method came out of that, and it took decades to realize and recover.

Seeing as the 'recovery' seems to be largely move fast, break things, worry about security never. I'd rather software not have recovered.

At enterprise level waterfall is pretty much the golden stardard.

Now that the agile wave has come and gone, we are doing sprint waterfalls.

A big reason for that is that projects, tasks and estimates are a hell of a lot easier to tack dollars onto than "sprints."

A big part of Agile is "We think this story might take x, but it can take x+y divided across 2n weeks of progress." Unless you have a PMO that can say "Okay, fine, but we're giving you $x to do it, manage it yourself," you'll always fall into waterfall because y is too difficult to financially forecast.

Rally by CA Technologies "solved" for this and gained lots of enterprise adoption somewhat quickly by trying to tack dollars onto sprints; from what I've seen, this essentially created a "scrumerfall" culture where teams have sprints with stories ("functional requirements") that come from epics ("design requirements"), but everyone has tasks and they'd better be estimated with hours.

This is what "Lean Enterprise" tries to address. PMOs should use data from their customers to drive their projects, and projects should be "invested into" instead of "estimated and rationed."

I don't know if it was the worst thing. I agree that it was unfortunate for the one reason you gave (waterfall), but there are other reasons that a new discipline(/profession) would want to emulate the practices of an older and more established discipline.

Another reason to choose "engineering" was to encourage software practitioners to incorporate some of the proven methods of traditional engineering. As an example, I've noticed recently that ethics training is an established part of an Engineer's education, but not so with Software "Engineers".

There is always room to improve, and I don't think ignoring the past, and justifying that ignorance by saying "but we're different" is necessarily the best way to achieve that improvement.

Weeks of coding can save you hours of planning.

Planning assumes you know what you're building. Most of my career has involved dealing with others who aren't sure of what they want, and whom change their minds on a regular basis.

See also: Big Ball of Mud http://www.laputan.org/mud/

Bam! You hit the nail right on the head. Now when its just a freelance customer its not as stressful, what is really horrid is when you have a product manager getting paid big bucks who has absolutely no clue how things work and no interest to find out, he has just one demand, "never tell me anything i ask for is impossible", that's a hard demand when he refuses to actually learn anything about technology or the web.

This isn't necessarily a bad thing. Software prototypes are an excellent tool to explore solutions to problems. We need to stop the hubris that we can immediately know exactly what we need without any effort.

Yes, but then exploring solutions should be an explicit goal. And the problem to be solved must be defined, along with criterias for evaluating solutions. Otherwise it's just an excuse for "we don't know what we are doing or why".

Agree. But then when the prototype becomes the product is when things really get bad.

But then it wouldn’t be called “hacking”, and so wouldn’t pass as software development these days. Instead, let’s have no planing whatsoever, no design and no requirements, have minimal time for actual development, “hack” code together and see where bugs take us. Aka “scrum”, “agile” and whatnot.

I feel like the problems you describe here (constantly changing requirements, minimal time for development, "hacking" code together) have less to do with agile methods and more with people-stuff.

I think agile methods can help a great deal in software development, especially breaking apart your projects into more discreet steps and supervising progress or giving the possibility for correcting course via sprints. Also using eXtreme programming tools like pair programming or code reviews to avoid having people slip into a frenzy of their individual domain / help keep obvious bugs out.

Having minimal time and changing requirements is just part of the game. And people "hacking" code together is just... well bad craftsmanship really

Agile methods are people stuff. Agile is just another management strategy.

Standard engineering design breaks projects into discrete steps, it's not at all unique to Agile. Doing that is essentially the definition of "work breakdown structure". Course correction time is built into Gantt charts. Engineers have had to deal with constantly changing requirements, minimal development time, etc, since long before software came around. They're further constrained because you can't just ship a patch to a bridge or plane after it's completed. And yet they manage to get reliable, functional work done!

Agile is management with ADD. Waterfall is a straw man process that no competent organization would use.

>>using eXtreme programming tools like pair programming or code reviews to avoid having people slip into a frenzy of their individual domain / help keep obvious bugs out.

Wtf does this even mean? What are you talking about. If the bugs are obvious why do you need extreme programming to find them? What frenzy? Extreme programming is garbage.

Agreed.

From my experience, when a proper preparation is done and there are clear requirements, most of the "obvious" bugs do not even come into play. They do, when "hacking" a feature and / or working in "POC" mode, which, for me, is not software engineering.

For me it isn't either and I'm certainly not advocating for "hacking" features together. The obvious bugs I mentioned were not design bugs, but simple slips that creep in when you've already worked for a few hours and your attention starts lacking.

>If the bugs are obvious why do you need extreme programming to find them

Because sometimes you've already worked for a few hours and end up wasting time with stuff, which could have been spotted easily by help of a fellow developer.

>What frenzy?

The frenzy I was talking about was referring to working on a project for weeks on end and forgetting that of course you are now skilled in that domain you built yourself, but at some point other people will have to work with that system too and for them your answers might not be that obvious. Having other people check in regularly can prevent this.

Also, please don't 'wtf' me, you could have just as easily framed your question in a more humane way. I wasn't trying to defend extreme programming here, but merely stating that a focus on open communication will lead to more robust software. Don't abuse my comment for furthering your personal hate against some software development methodology.

sorry fren

Do you ever get the feeling that software "engineers" are expected to be little more than business-ops guys with a higher IQ and a habit of unusual fascination with solving problems that must be done in tiny detail?

Don't worry about creating bugs, we have a separate team to fix those.

Having recently made the mistake of diving too quickly into coding something, without having done enough analysis, I have to upvote this.

Sometimes nothing slows you down as much as being in too much of a hurry.

On the other hand, writing testable code can be a perfectly valid and very useful part of the design process - even at its initial stages, when things are just being explored. There is nothing special about writing code compared to scribbling on a piece of paper or just walking around and thinking. Who says that pseudocode must always be written in a pseudo-language?

Incidentally, prototyping also gives rise to a very powerful idea of solving the problem by first creating a domain-specific language in which a solution to the problem can be expressed in the most natural way, and then providing an efficient implementation of that language.

Yes, and often diving into the code works for me, which is why I'm in the habit of doing it, but in this case two or three days of whiteboarding and analysis might have saved me a couple of weeks of going down a dead end.

Haha, you just summed up perfectly what I was trying to describe with my child-comment to another poster on here!

There is also the opposite, which is even worse: Weeks of planning can save you hours of coding.

I wear this shirt to work.

"Whither Software Engineering?", discussed in the OP, sounds like an interesting historical piece about how long it took for mathematics to be integrated into engineering work. The author is John Allen, who wrote Anatomy of Lisp.

Can anyone find that paper online? https://news.ycombinator.com/item?id=962733 points to a blog post about it from 2009, but none of the links seem to work.

It's here: https://pdfs.semanticscholar.org/3a6b/0506b8b922d7e6cc933a39...

You can find papers more easily one "Google scholar", I found it there:

https://news.ycombinator.com/item?id=14528514

Ah of course. Thanks!

FSCQ is a really great example of a large system with proofs of correctness using extraction from Coq.

Another well known project is CompCert the certified C compiler [1]. Which has seen a fair amount of external testing and use in verification of GCC and Clang as a reference for checking invalid compiled semantics [2] (to say nothing of compiling programs).

1 http://compcert.inria.fr

2 https://blog.regehr.org/archives/1052

The problem with this line of thinking is that a piece of software is ultimately built to solve a problem in our real, messy, uncertain world. There is no way to "prove" that an application served its intended purpose. As Fred Brooks wrote: "Even perfect program verification can only establish that a program meets its specification. […] Much of the essence of building a program is in fact the debugging of the specification."

Exactly by the time you could formally prove the application is correct, the market will have moved on to something different. Right now the combination of quick to develop and mostly correct seems to be more desirable to companies then slow to develop and formally correct. The success of languages like JavaScript and python show this.

So what would such a curriculum look like? I've tried diving into this a few times and it often feels like hurling yourself against a wall in hopes of picking up bricks. You actually do make progress after a time but it's difficult to determine the curriculum flow.

For me I think it started with reading a throwaway aside on "type verification" and wondering what the heck that was. After a bunch of wikipedia searching into different types of type calculus (not fun), I saw a reference on dependent types, which led me to Coq tutorials, and then, in an effort to be more practical, learning more Scala and a some Haskell - reasoning being that even if they didn't have full support for dependent types, at least their type systems were on the right track.

Failed efforts so far have been learning TLA+ and Idris (I haven't yet found a way to apply any learning), and trying to get through some youtube lectures on homotopic type theory. Maybe next time.

I don't know, maybe something like this, degree in informatics engineering (sorry Google translate):

https://translate.google.com/translate?sl=pt&tl=en&js=y&prev...

Check the "Consolidation Units" and "Specialization Units" required for earning enough credits.

Also the degree is validated by the Portuguese Engineers Society.

Just one example, most Portuguese universities offer similar degrees.

I am a strong proponent of Engineering in our work and how software should be under the same quality regulations as other products in general, not only when there are human life's at stake.

Looks a lot like mine (also western European but with more sysadmin topics and less graphics and AI).

Luckily someone has paved the path for you :)

https://functionalcs.github.io/curriculum/

There exists a body of theory about how to do software engineering as described here, but there doesn't exist a sizable community of software engineers willing to do this kind of work on a reasonably large scale. Judging by HN (which itself probably represents the upper-end of 'engaged' software developers), most people fidget uncomfortably when forced to explicitly declare derived types, and gnash their teeth when faced with Haskell monads; who among you would make a career out of writing Coq proofs about system correctness?

Optimistically, the community might come around to realizing that this sort of development is at least desirable, and we might wind up with the basic components of popular OSes, or of firmware for dangerous things like cars and medical devices, being produced via provably-correct processes. But that will almost certainly wholly exhaust humanity's reserves of strongly math-educated software engineers (not to mention, industry's willingness to pay for it.) Software engineering may be impossible because most humans don't like math.

> there doesn't exist a sizable community of software engineers willing to do this kind of work on a reasonably large scale

also on the other side there's a very limited pool of employers willing to pay for this kind of engineering - aereospace and military mostly, and the second is cheapening out too.

I'm here for it, and I know a couple hundred working programmers personally who are with me.

I think a significant issue in software is the comparatively unconstrained input space.

I would imagine the average natural gas power plant would have more than a few issues if you fed it pure hydrogen instead of whatever methane based mixture it was designed for. Similarly I imagine there are other inputs that have tight specification, lubricants, etc. But, these inputs are relatively easy to control via both human and machine processes and there aren't that many combinations you would consider valid. There are complex inputs in the physical world, for example the impact of complex wind patterns on bridges, but I still feel this is simpler than many input spaces in software.

In software, at some level you're almost always dealing with human or machine generated input which may be very complex in nature such that that even after all reasonable efforts at validation the size of the possible input space is massive if not effectively infinite. We're getting a bit better at dealing with this, e.g. fuzzing, but there is a long way to go for fuzzing to be viable on more complicated inputs.

Computer Science is a science field, therefore computer scientists usually see themselves as something more than an engineer - you don't want to apply the same knowledge over and over to make something exactly as the known knowledge tells you to do, you want to try new things and experiment with new ideas. That's the general mentality of the degree by itself.

Real "software engineering" would have to be done with Agda, Coq, or similarly tools - theorem provers that guarantee your solution is sound. It is done that way for software that absolutely cannot fail (eg, aerospatial). Unfortunately, engineers are usually too far away from this kind of knowledge - and sometimes even doubt this is possible or doable, even if you direct them to the research (!).

By the way, software engineering disciplines are usually depressive OO garbage, frameworks and all that non-sense that comes with excessive Java usage, and frankly never teach anything about reliability or even proving programs correct - which should have been their primary goal.

Having a degree in informatics engineering I strongly disagree with that, given the lectures we had as part of our curriculum.

When you build a bridge, a power station, or a wind turbine, you're creating something of known economic value for several decades, even centuries. The whole process is expected to have little to no business deliverables for years. That lends physical engineering nicely to an established set of standards. In software, we're often innovating entirely new products, and people want to see working proofs of concept, and see organic growth with the user base without getting locked in to preexisting concepts of what the product should do the business figures out what the market is.

The nexus in products like vehicular and medical equipment software might be a good place to start implementing something like a discipline, but those systems are so much more singular and simplistic in their product function (not the implementation, I realize) than what the average software dev is working on that I don't even know how much downstream influence it would have.

Your hot new gamified Pets.Com for mobile doesn't need to be engineered. A web server handling payments absolutely should be.

Nice title, although I don't think the proposed solution (formal proof methods) is the answer. My own view is that building software is more of a design process than an engineering process, for the simple reason that software rarely has the kind of fixed requirements that make engineering solutions possible. Most software is constantly evolving and the factors that make it possible to evolve well include things like conceptual clarity and modularity, about which formal methods have nothing to say.

There was a movement around 20 years ago to establish a field of practice called"software design" but I don't think it went anywhere. The problems remain the same though. http://hci.stanford.edu/publications/bds/

There is a movement going on from weak typing back to strong typing which is the same idea approached from the opposite (practical) end. Have the computer do more work to verify program correctness by rejecting invalid expressions.

Maybe the two will meet at some point?

I never got the excitement that some of my college classmates had for JavaScript because they didn't have to 'deal with types'. I always much preferred a compiler telling me that this won't work instead of having to find some obscure type bug at runtime. Don't get me wrong, I still very much liked programming in JavaScript for other merits, especially the functional aspect or flexibility of extending your program simply in a browser command line window.

EDIT: I kind of got distracted by my JavaScript-story here. What I originally wanted to post was: I always had an interest in philosophy and the logical thinking you can apply in software development (i.e. you can save yourself so much trouble by just thinking through your system thoroughly before writing even one line of code). However, the general sentinment that I also got from a lot of my classmates was that they didn't want have anything to do 'with any of this stuff' (this stuff being the humanities) but didn't have a problem with cracking hard math problems day in day out (which for me at least is pretty similar to using logical thinking / philosophy on a less abstract level).

Sorry, if by my large edit I confused some people!

You wouldn't be alone as a philosopher-developer, there are a decent number of us out there.

However - shipping code is often very organic, cultivated stuff which foils attempts to establish a coherent philosophy, a through-line of "how it works" - it works because it worked the day before, all that changed was that a little bit more was added. Repeat till broken, then refactor back to sanity(else abandon ship). It's Sisyphean hill-climbing.

This in turn plays into the falsity of "engineering" as a way to describe what's going on with a lot of code. You can do it in the small, but at any scale the codebase habitually becomes a living organism. And that tends to get puzzle-solver types excited, because if it's alive and keeps changing, then they will have endless problems to solve forever!

So I believe the philosophers of the crowd ultimately tend to move away from the applications coalface and look for something relatively smaller that does allow some time for reflection and distillation of the problem.

I find your last paragraph very interesting. May I ask, did you make that move yourself and, if so, where to?

In my daily work I often refactor an old, organically grown, codebase and that's exactly what I do:

I try to think of what the system actually is / should be and try to put this in code in as concise terms as possible (that's kind of a general description of software development, but I just wanted to contrast this to simply hammering in the quickest fix you can think of). A lot of times I wonder though, if that is really what is needed and a simple 'prototyper' kind-of-developer would get the job done faster, although maybe not as elegantly (slight humble-brag there).

You should look into TypeScript if you haven't already. It's not perfect, but I think it strikes a nice balance between those two approaches. It's my "go-to" for new frontend projects because I enjoy the compiler's flexibility in regards to type checking. I can explore alternate approaches to solving the same problem (and by proxy, tests) and gradually introduce stronger compile-time checks as the solution comes together.

Thanks for the input. In my place of work, where I mainly use C#/.NET, we are in the process of planning to implement a new frontend for one of our services. I was totally looking forward to using this to get some experience with TypeScript!

It's a question of balancing planning with acceptable ambiguity. Types add compile-time assertions but the cost is a constant mandatory overhead that still requires behavioral testing. For my space, I prefer well designed systems and TDD/dynamically typed code.

However, I absolutely agree that "systems thinking" is seriously underemphasized in undergrad. Bad algorithms are replaceable, bad systems require deeply invasive refactors.

To some extent, they have. The big breakthrough was "auto" in C++. (I wanted to call it "let", but that involved introducing a new keyword.) The type of a local variable in C++ is quite verbose for iterators. This discouraged the use of iterators in FOR statements. With "auto", you can write

    for (auto p = thing.begin(); p != thing.end(); p++) { ... }

for anything that uses an iterator. No need to write out the whole iterator declaration for p. Or to generate it in a template. Suddenly, iterators became far easier to use.

Since then, Go and Rust have taken a similar approach of limited type inference. There seems to be a developing consensus that function parameters should have explicitly declared types, but within a function, type information should be inferred when possible.

We're seeing convergence from the other direction, with Python and Javascript (via add-ons such as TypeScript) acquiring type declarations.

So the two are meeting.

For me the biggest advantage of strong typing is ide quality that follows - refactoring support ans code completion are much much better for example. Makes work on larger project significantly easier.

My sense is that it won't really get close to meeting until dependent types are more mainstream. Idris is out there but not sure how usable it will end up being. Haskell and Dotty seem to getting closer even though I don't believe they entirely support dependent types.

The two have already met. In Python you have optional typing [1] . Or in JavaScript you could use Flow [2] for static type checking. Also you could interface JavaScript with Typescript [3]

[1] https://docs.python.org/3/library/typing.html

[2]https://flow.org

[3] https://en.m.wikipedia.org/wiki/TypeScript#Compatibility_wit...

Python itself does not have optional typing, only optional type hints.

You need additional tools for the type checking, and these tools are in experimental stage.

Although you make no citations of your claim, I think you mean a trend towards static typing over dynamic typing. Strongly typed languages are already pretty common.

No, I meant what I said. I'm talking about mindshare, with e.g. typescript gaining ground on javascript, or php 7's strictly typed style gaining ground over php 5's loose style, which is all in the dynamically typed camp.

TypeScript brings static type analysis to JavaScript, not strong typing. I'm not sure what you mean by PHP 7's "strictly typed style" but I assume you mean its type annotations, which it's had since PHP 5. Those are neither a case of strong typing (PHP is duck typed) nor static typing (since it's evaluated at runtime).

This convergence (deliberate or not) has been noticed.

I've seen a diagram like this http://wordaligned.org/articles/programming-nirvana-plan-b but with more detail. ie. with C++, Java, Python placed in there somewhere.

(probably in the linked paper, but I can't access it now)

So....Medium typing? Would that just be all the inference of Python mixed with the literalness of Swift?

Actually I meant that the formal proof to code and the strongly typed code camps would eventually meet, as the type system grows richer it may approximate a proof.

The Homotopy Type Theory folks are way ahead of you on that one.

Teams that are 100% invested on having clean, tested and reliable code can have software engineering, absolutely.

The problem is that outside of the tech industry, most of the people with the pursestrings don't see the value in it most of the time.

Why? When their main widget makes $100M/month (for example) and its current maintenance, as fucked as it might be, is ($1-10M/month, or 1-10% of revenue), spending, say, $1-10M to drop that OpEx to half of current costs, prima facie, doesn't make a lot of sense.

What changes that equation is when some event that could have been prevented by good software engineering principles occurs. Good examples: the latest British Airways IT meltdown, LinkedIn's massive data breach, 100's of millions of CC's stolen from TJ Maxx and others.

You see, when something happens that threatens that $100M/month cash machine, many millions of dollars of bonuses, raises, hirings, parties, expansion plans, and other general symbols of growth and progress get threatened as well. THAT is what people with the pursestrings don't want to see messed with.

The overall problem with this is that it's reactionary, and every reactive event has a lifespan. So it takes finding someone with connections that can either capitalize on an unfortunate event trailblazing a path for software engineering to happen, or tell a good enough story of previous woes and misfortunes that can make that happen all the same.

TL;DR: If you want "software engineering" to happen at your company, you need to tell a compelling story of how NOT making it happen will lead to great losses, and then you need to be extremely patient.

I've looked into Coq and like a lot of others here tried to learn the theory and application but it is so far removed from my daily work that the benefits are basically non-existent. I've gotten way more mileage out of trying to learn systems theory and cybernetics than anything related to dependent types.

People think this is a tool problem but it's not. Well, the tools kinda suck but fundamentally it comes down to culture and mind share. My background is in pure math but my programming knowledge is steeped in the hacker tradition. It is very hard to overcome all that momentum.

There is no killer application for verified programming. Whatever you can do in Coq in 10 weeks you can do in C in 2 weeks with a few extra buffer overflows. We'd be further ahead if there were more tools for validating C code than trying to rewrite the world with dependent types.

You're right it does come down to culture and mind share. That's why the author already admits this would be a long process. But I think he's right and to truly call what we're doing software _engineering_ there needs to be a more formal basis for how to do things and especially to discern between this is right and this is wrong.

No offense to you, but often times people that entered programming originally from related fields and taught it themselves give me the hardest time working with them. Like, no it's not enough that the software now provides this feature but you copied 1.200 lines of code for it and just squeezed in your if-branches and for-loops somewhere. Your job as software developer is to think how you can wed these new requirements with our existing codebase for others to build upon.

Sorry, but as a bit of a pedantic and clean code enthusiast this is kind of a passion topic for me :)

There’s a terrible trend in the San Francisco tech scene where “engineer” is viewed as a synonym for “software engineer”, which is itself a synonym for “programming”. It isn’t.

That said, I tend to disagree with people who believe the solution is "use Coq".

We have existence proofs that Software Engineering is possible. For example, look at what NASA did with the Space Shuttle flight control software. But for most business domains, real engineering simply isn't cost effective.

https://www.fastcompany.com/28121/they-write-right-stuff

I'd like to contribute a definition of the words "engineer" and "engineering" from an old dictionary I had lying around.

From Webster's (1948):

engineer: n. 1. [Rare], a person who makes engines. 2. a person skilled or occupied in some branch of engineering: as, a mechanical engineer, an electrical engineer. 3. the operator of an engine; especially the driver of a railroad locomotive. 4. in military science, a member of that branch of the army which is concerned with the construction and demolition of bridges, roads, and fortifications, the laying and sapping of mines, etc. Abbreviated E., e., eng., engin., engr. v.t. 1. to plan, construct, or manage as an engineer; hence, 2. to contrive; manage skillfully; superintend; guide (a measure, action, etc. through).

engineering: n. 1. the planning, designing, construction, or management of machinery, roads, bridges, buildings, fortifications, waterways, etc.; science, profession, or work of an engineer: abbreviated E., e., eng., egin. 2. a maneuvering or managing.

Feels like some stability in computing generations (CPU/GPU) or interaction paradigms (mobile) will be necessary to formalize software engineering. Until bridges/etc had a relative standard building each one was a craft. Not to say there hasn't been generational tech in traditional engineering, but generally not on the order of every few years.

It'll help, but it's not essential. There are practices that we can adopt today that will formalize software engineering. We should look to systems engineering and modeling (as used by other engineering disciplines: prototypes, mockups, math models, etc.). We already have many of the necessary formalisms, either from the fields of Math and CS, or from other engineering disciplines (notably systems engineering).

The problem for software engineering is that it's really easy to model with code, and those have an unfortunate tendency to become the final deliverable. We have short iteration cycles that make the benefit of systems engineering and large scale engineering approaches less obvious. In CE it's quite obvious that a model of a bridge (physical and mathematical) will have benefits towards the final bridge construction. It's also obvious that it's just a model.

In software, the model is too often functional enough to pass as the real thing, but fails in various ways: was constructed haphazardly allowing security vulnerabilities or stability issues; wasn't designed to run quickly or to scale effectively across multiple servers; not developed with maintenance in mind.

I think it's probably not possible. Here's a way of thinking about why that may be: what we're doing when writing software has as much in common with doing science as it does with engineering. By 'science' I mean coming up with a formal description of how some physical system behaves. If we're willing to generalize this activity to include non-physical systems then, I'll argue, software development would be included as the same kind of thing.

Fundamentally writing software is about constructing a formal description of a less formal 'specification' of some abstract system. The architecture used in this formal description corresponds to what's ordinarily called theory in science; when we execute our formal description, it corresponds to performing an experiment, which may lead to the discovery of anomalies (i.e. bugs); sometimes this anomalies are significant enough that we are led to revise our theory (i.e. refactor); if our re-conceptualization is significant enough and we decide to use an entirely new architecture, you're running into something like a paradigm shift with all the resistance and other social factors this entails.

Coming up with architectures/theories to formally describe systems in the foregoing manner is not something we know how to make reliable. If that goal is clearly out of sight for science, why would we expect it to be achievable in the sibling discipline of software development?

The exception I see to this (which admittedly is a large exception) is that probably most programs which need to be written are very similar to programs that have been written before. If we had some unified process for evaluating architectures for these programs that we write over and over again, then we'll probably eventually stabilize on 'true' architecture for that class of program. We see this happening to some extent in the creation of 'frameworks,' but maybe we could come up with better measures for comparing the aptitude of competing frameworks?

It could also be that the physical systems described in science tend to be amenable to simpler description than typical software specs, which may be loaded with non-unifiable features. In this case, frameworks/theory only gets you so far because most of the work is just describing special cases...

Anyway, maybe the 'science' we need to found software engineering on top of would be a formalization of this general process of coming up with formal descriptions of either abstract or physical systems. I bet there are regularities there which could be exploited for systematization—and if nothing else, there are likely insights to be had just through seeing software development as a thing having significant relations to the practice of science.

Edit: phrasing, typos, etc.

I think really hits the nail on the head. It also may be the foundation for why we find things like neural networks more usable for things like vision than directly formalizing the approach. In other words, you could use an approach like coq to verify the neural net software calculates correctly, but you couldn't use to formalize what it calculates.

You've put into words better than I ever did.

I hope things go this way. I'm betting my future on it.

The best way to predict the future is to invent it ;)

Good luck!

In my opinion, "Software Engineering" has two main problems:

- Incompetent management.

- Incompetent programmers.

Incompetent management, because of "sky is the limit" effect, pretending being some Steve Jobs or similar, instead of doing the real job: risk management, resource allocation, put competent people in charge, and being involved so the project(s) are kept on track.

Incompetent programmers, because not being qualified enough, not having enough experience, or whatever reason making the programmer being nihilist, or a mix of the previous.

If you have competent management, and competent programmers, you'll deliver. The problem comes when you have incompetent and/or corrupt management, taking decisions from emotional impulses, or thinking in their bonus first, hiding the problems, and making the "bomb" explode years later. Then, you'll receive the excuses: legacy code, architectural problems, blaming contractors, processes, methodology, etc.

No, it's necessary

Reply on news.ycombinator.com