Why is there so much useless and unreliable software? | Hacker News Reader

Why is there so much useless and unreliable software?

83 pointslearningstud144 comments

Linear logic has been known since 1987. The first release of Coq (dependent types for functional programming and writing proofs) was in 1989. The HoTTBook came out in 2013. Ada/SPARK 2014 came out the same year as Java 8 did. We also witnessed the Software Foundations series, the CompCert C compiler, the Sel4 microkernel, and the SPARKNaCl cryptographic library.

Instead of learning about those achievements and aiming to program for the same reliability, clarity, and sophistication, we see an abundance of software that cannot clearly describe their own behavior nor misbehavior.

Instead of incorporating the full functionality of XML/HTML/CSS/SVG/JS/WebGL into the development experience and providing ways to control them at the fundamental level, we reinvent crude approximations like the various web frameworks.

YAML and JSON often trumps XML/XSD until things get out of control, and even then, people still don't learn the lesson. Protobuf, flatbuffer, capnproto, and the like keep reinventing ASN.1.

Naive microservices partially reimplements Erlang's BEAM VM while ignoring all the hard parts that BEAM VM got right. Many people riding the microservice bandwagon have never even heard of Paxos, not to mention TLA+.

Many programmers keep learning new shining frameworks but are reluctant to learn about the crucial fundamentals, e.g., Introduction to Parallel Algorithms and Architectures, nor how to think clearly and unambiguously in the spirit of Coq/Agda/Lean.

No wonder ChatGPT exposes how shallow most of programming is and how lacking most programmers are in actual understanding. Linear logic and dependent types are there to help us design and think with clarity at a high level, but people would rather fumble around with OOP class hierarchies (participate in the pointless is-a/has-a arguments) and "architecture" design that only complicate things.

What is this madness? This doesn't sound like engineering.

View on HN

> Why is there so much useless and unreliable software?

https://twitter.com/maxkreminski/status/887815522061926400

"a reminder: if inexperienced creators are using your tool to churn out loads of half-baked garbage, your tool is a phenomenal success"

Software is such a powerful concept - it basically imbues physical objects with magic - that even bad software is hugely empowering to its users and takes off very quickly. The demand is staggering.

I appreciate that when you see yourself as the most intelligent person in the world it becomes intolerable to be surrounded by unthinking muggles scratching in the dirt, but after a while you realise that life is more complicated than that, people usually have good reasons for doing the things they do, and that perfection is neither attainable nor necessary for most of that.

It's never about perfection, but lifting up the acceptable lower bound. Around 150 years ago, Joseph Lister had to go to great lengths to persuade surgeons to apply antiseptics, which seems like a no-brainer nowadays. The methods I mentioned have been well-established for decades.

Generally, nobody forces you to use software you don't consider reliable. That's especially true if you use open source software (and if you don't, you've got other problems).

That is technically true but in practice one has to use certain software to some degree or at least faces consequences of some software's output. Bad software has sent innocent people to prison, declared some officially dead etc without those people ever having a say in what system should be used.

Unfortunately high quality software tends to be so much more expensive in various ways that almost nobody bothers. Which is understandable, but in some cases still unacceptable.

Don't get me wrong, I'm not talking about open source software, those developers don't owe me anything and often produce better results anyway.

I agree that in some areas the bars could and probably should be set higher. While in most domains there are strict certifications (e.g. cryptography, military, banking, aviation, medical devices) there are certain grey zones in which unreliable software impacts our lives. Ideally, these gaps are closed. For example, EU legislation requires a certain amount of transparency for a system a citizen is subjected to, e.g. according to the GDPR customers have a right to know why they were given a low credit score.

It's a double-edged sword because certifications are complex and sometimes expensive (e.g. FIPS certification for encryption), so smaller software companies have a hard time competing, potentially leading to more quasi-monopolies and cartels.

"Acceptable" is a transitive verb. Acceptable to whom?

And a more subtle question: what happens below the acceptability theshold? It doesn't seem that acceptability can be driven up costlessly at zero time investment, so what this means is either "less software is produced" or "software is more expensive".

(We see a version of this with the app stores, which enforce a really crude "quality filter" by banning apps)

"Acceptable" in not even a verb, never mind transitive.

Surgery isn't learned at home, the cost of failure is so high that a certain skill level is considered the minimum.

Though if you're talking about the lower bound for calling one's work "software engineering" you have a point, I think it's reasonable to expect some best practices from experienced developers. One problem seems to be how easy it is to underestimate the cost of bad software and what it takes to produce good quality. This reminds me a bit of the wildly inconsistent quality of CGI effects in today's movies, which is often the result of late changes and in turn insane deadlines, to some people VFX is magic anyway so why can't it also magically be done faster?

You could say the same thing about most things. A "bad" chair from ikea empowers people to not sit on the floor. A "bad" meal from the local cafe is still nutritious and tasty even if it's bad compared to something 5x more expensive.

Yes. Although the original concept was about tools-for-tools, specifically various sorts of "game maker" software. This stuff is so "bad" that OP wouldn't recognize it as software .. except that it allows people who wouldn't otherwise to make and share games, having fun and enjoying creativity.

On one hand I want to agree with the quote you posted, but on the other hand implying or making it seem like the OP sees themselves as "the most intelligent person in the world" is putting the question into a wrong light and seems disingenuous. The OP's question is valid and one should be allowed to ask it.

Most tech choices are made on the basis of incomplete information and incomplete planning or even no planning ahead. Many are merely following hype, instead of truly looking at the options and making a wise choice. Hype creates awareness of products or software, which ultimately reaches the uninformed masses. People not in the business of making software themselves have usually vastly less information to base their choice on and often make questionable choices. This in turn generates more incentive to continue making software like the one they chose. This is what amounts to the quote you posted.

As a software agency owner I can tell you that barely anyone wants that. We maybe get 1-2 requests for quality software every year. Most of our clients want their apps built as fast as possible, with little to no consideration about the tech quality. We have the choice of doing it that way or going out of business.

This also affects our people (usually our top engineers) - which is why I want to start developing our own products this year.

I can relate. So often the people in charge doesn’t care at all about software quality. They want things as fast as possible in order to make money out of it. Software is just a mean to an end. So rarely an end by itself … Try to build a car or anything half as fast … control quality will suffer but with software it’s way easier to ship fast

Ding, ding, ding!

Cover your ass, ticking off items as launched are other areas that lead to speed over quality every time.

The OP post reads as very naive, without experience in the real world corporate politics. No one really gives a shit about most things on that list (unfortunately).

Same here. Some projects I work on:

- please put this broken crap of 15 services written in 6 different programming languages on a 50 node cluster so we can read and write data with 300 kbit/s.

2 years later:

- could you do something about this cluster, it costs us too much.

Some companies may not have even existed 2 years later if they spent the money and time earlier to optimize.

We are talking about 10+ years old companies. I quite often save significant amount of money for such companies which could have been saved from the beginning by not doing something completely silly.

Yeah, then when the half baked thing you've rushed out falls over it's your fault. Despite the customer asking for exactly that level of quality. When bugs appear it because the engineer is an idiot not that they wanted to write tests but no one wants to pay for them. Agency work is tough. Being an in house dev is much better IMHO.

It's the same for most industries. I constantly see comments from construction workers "Every building I worked on was rushed and absolute crap, I wouldn't be surprised if they collapsed in a year" and yet they almost never do.

>It's the same for most industries. I constantly see comments from construction workers "Every building I worked on was rushed and absolute crap, I wouldn't be surprised if they collapsed in a year" and yet they almost never do.

Its because they are engineered with a margin of safety and then inspected for obvious issues. But yeah throw them together is the name of the game. Faster = more money.

Take the aftermath of the recent Kahramanmaras earthquake, the status quo should be improved even for well established engineering fields, and more so for the software industry.

Even with own products it often is an issue; after years of working on something complex, you want or need to launch it. But it might still be quite bad compared to your vision of it; you simply run out of time, steam, money etc and launch it. Tests succeed and with ok coverage, you even have some tla+ and coq covered parts, but, because you have to be somewhat practical you stopped proving things (or even just dropping some less uphill practices) years ago.

I think it boils down to that many people are eager to start projects but can't be bothered to fi

I think you are

I think you am

Consider it do

Is it just me or has HN's meme-fication exploded recently? Only seen stuff like this on Reddit before. I chuckled though.

I think these pointless threads even get deleted, not just collapsed. So you are just here too early.

Thank god. One of the things that make reddit unbearable.

Engineering is about tradeoffs.

When you're shooting a multi-hundred million satellite into orbit it's worth the extra few million expense of formal verification because otherwise you might lose a gigantic investment and even kill people in the process.

When you're working on something without such extreme constraints, with vague SLAs, and limited to no business risk, then regular unit/integration/etc tests are good enough and exceedingly cheaper.

> When you're shooting a multi-million satellite into orbit it's worth the extra few million expense of formal verification

And even then the cost effects of that are so prohibitive that SpaceX transformed the industry with their "we cannot guarantee the hoverslam landing will work first time" approach.

Producing something imperfect quickly and then iterating beats upfront planning by such a large margin so often that it's not funny.

Engineering is about tradeoffs.

People can give themselves all the fancy titles they want; I don't think 99% of software development is anything approaching Engineering.

(This isn't so much an attack on my fellow programmers as a recognition that this field is very young and still very immature.)

Not taking it as an attack as I don't like to be called a software engineer either, but the definition of engineering definitely applies to software engineering as it is right now:

> engineering; the discipline dealing with the art or science of applying scientific knowledge to practical problems

We do apply scientific knowledge to practical problems, all compsci material of data structures and algorithms is science being applied to practical problems.

Do you know the difference between an engineer and a tinkerer? The engineer can lose his license for negligence

Every engineer I know has heard something like that, and knows that it applies to them. Except for a specific kind of engineer, and I bet you can figure what job title those engineers have

We do apply scientific knowledge to practical problems. The discipline isn't there, however, for the vast majority of us. In fact, most companies prefer it that way because proper design creation/validation takes a lot of time

Changing a tyre does not make you an engineer. I mean, half of us consider the idea of having written requirements bollocks that only exists to make managers happy, and half of the rest are happy to trudge along without knowing what actually needs to happen

Calling what most of us do "engineering" cheapens the meaning of the word

> Do you know the difference between an engineer and a tinkerer? The engineer can lose his license for negligence

The engineer that is part of an association of engineers, working in one practice of engineering that's under a licencing scheme can lose their licence, that doesn't mean that engineering is only that... You're pigeon holing engineering to only the practice performed by licenced professionals, which is not true for the meaning of "engineering" as a whole so I don't agree with your premise a priori.

> We do apply scientific knowledge to practical problems. The discipline isn't there, however, for the vast majority of us. In fact, most companies prefer it that way because proper design creation/validation takes a lot of time

And those companies don't have a mature software engineering process, throughout my 20 years in software I've worked in them (startups usually).

On the other hand I've also worked in a couple of global tech companies that do apply software engineering practices, that do designing and RFCs to gather data and feedback about the proposals. That do apply processes for software design and architecture to maintain those systems healthy through a 5+ years timespan, etc.

Other companies don't need or want the heavyweight all of this processes carry, and they might be right in doing so if their systems won't cause major pain for a large swath of the society, they can tinker with their systems and play around, I agree that it isn't hard engineering but... Is it needed for these cases? Does everything built with software need all these practices or can these practices be used by the professionals and companies that require them?

> Calling what most of us do "engineering" cheapens the meaning of the word

Let's agree then that larger orgs do some kind of software engineering at least, a lot of smaller companies are still in the tinkering phase and that is completely fine.

>You're pigeon holing engineering to only the practice performed by licenced professionals

It's a joke that most of us heard in one form or another back in Uni. I'm sure it's been repeated for centuries before the word "software" was first uttered

But then we all were studying some kind of Engineering, and only a very specific subset of us expected not to need a license after graduation... funnily enough, only that same subset didn't have to learn the core subjects that "all" engineers have to learn (tee hee)

>which is not true for the meaning of "engineering" as a whole so I don't agree with your premise a priori

As long as you are happy to call duct taping some damaged mechanical part "engineering" too, at which point why bother with the expression?

>I agree that it isn't hard engineering but... Is it needed for these cases?

I agree. But then I don't feel the need to call it engineering when it has very little to do with everything else goes by the same name

>a lot of smaller companies are still in the tinkering phase

A lot of big companies are still in the tinkering phase too - I've briefly worked for a certain big bank, and I live with someone that works for a certain health org that moves tens of billions of dollars every year. Their level of engineering is a joke too

The discipline itself isn't mature enough to warrant the name engineering. Additionally, it's incredibly amusing how much resistance the actual practicing professionals have against changes that would nudge the discipline towards warranting the name

We live in the wild west, act like cowboys, want to continue acting like cowboys, have the responsibility level of cowboys, and expect to be called engineers. Presumably because being called an "engineer" makes you sound legit

that does not make it less about tradeoffs. perhaps even more so?

For context: I come from a mathematical background. I didn't particularly appreciate that most programming languages use the word "function" for something that is not a mathematical function (e.g. has side effects). Well, then I got used to it that sometimes it is practical.

> This doesn't sound like engineering.

It is precisely engineering. As opposed to pure science and art. (I consider mathematics to be, above all, art.)

Based on the post itself, you come with a theoretical mindset. You may consider purity to be more important than practical applications. Yet, if people write software to be used, they focus on the latter. Sometimes it results in hacky code even within an already hacky language. There are no extra points for purity.

Purity itself is a double-edged sword. Sometimes it makes the code more reliable. Other times - it generates a lot of abstract nonsense, which makes it harder to reason about the piece of software or change it.

On the positive side, look at the Rust language (and community!). While it has lovely abstractions and safety guarantees, it is a practical language - performant for writing and execution.

Rust is pretty hard to write, its not "practical" in any sense

Practical as there is already a lot of software written (or rewritten) to Rust. Sometimes for safety, sometimes - purely for performance. I don't claim it is an easy language or a universal solution.

Compare and contrast with pure languages that are less versatile. Sure, Haskell has its practical uses. From a popular example - Pandoc. Yet, I know many programmers who have been preaching it is the best language but could not write a performant program that solves a given task that does not focus on algorithmics.

By contrast, PHP or early JavaScript were not pure by any sane standard. Yet, they conquered the web. (Fortunately, ES6 and TypeScript made the JavaScript environment much saner.)

It has multiple safety-release-valves, programmable builds and full blown procedural macros. It's very practical. What it is is "unwieldy", as in, annoying to use.

"Rust is pretty hard to write" is an opinion.

FWIW, I happen to have the opposite opinion. I find Rust easier to write than, say, Ruby or JS, because it is statically typed, has enums and great pattern matching, and traits are great.

Rust can be cumbersome when you don't want to worry about structure and design beforehand, yes. But for writing reliable software, I think it's actually the other way around - I prefer having warnings about `unwrap()` over hunting down a missing null check or typo in a variable name.

It is a function that maps global state to global state!!

So is the Universe, when sliced by time.

Actually probably not, since clocks are always local

We can replace 'time' clocks with vector clocks ordering causality.

Why? Because of the barely competent programmer, like myself. A DIY army.

Is XML better than JSON? For longterm stability sure. A quick config file? Nope. You see, to understand XML you kind of have to know how to work with trees. Bread-and-butter for compsci grads, a nuisance for all others.

Recently I needed to query a SOAP-based API. It took me 3 days because I had no idea how a namespace in XML worked, because I was not sure which lib in Python to use (lxml was the solution), and because their API had some quirks. I read forum posts from the late 00's, the documentation is really scarce.

I would love to learn the fundamentals, I don't like half-assing things. But a) I'm not going back to uni for it, b) I'm not bright enough to be excellent in two fields. Learning the fundamentals takes a very, very long time. Where to start even? A complete math program? Programming fundamentals as such? Theoretical compsci?

At the end of the day, things need to work. If they fall apart next year I can tell my boss "ooohhh big problem give money". Using XML instead of JSON has no payoff today.

This is exactly what OP is talking about. You have XML instead of ASN.1 Imagine if we had only one way of doing data transfer between computer systems.

> Imagine if we had only one way of doing data transfer between computer systems.

Well, that would be deeply inadequate and people would have to invent more of them?

I am curious what use cases triggered the development of XML for example.

This roughly covers it: https://ccollins.wordpress.com/2008/03/03/a-brief-history-of... - I hadn't realised SGML was quite so old, though.

> At it’s core, XML allows a software engineer the ability to create a vocabulary, and use this vocabulary to describe data.

The lols. This is so good to read that XML allows that.

> Is XML better than JSON? For longterm stability sure.

Don't get that.

>to understand XML you kind of have to know how to work with trees.

JSON is trees, too.

Am I the only one seeing the forests here?

> Is XML better than JSON? For longterm stability sure. A quick config file? Nope.

I have an opposite opinion. JSON is great solution for a protocol or serialization format, better than XML (JSON is faster, smaller, safer, easier to understand) But it is one of the worst possible choices for config file format. Even XML is better in that case. Lack of comments is complete showstopper in any source code file syntax.

Nice rant but the answer is simple - incentives.

Seems like there's not that much demand for what you would like to see.

Crappy, barely working, software now is better then 1 year delayed perfect one.

I agree that commercial incentives are not there, you must always ship earlier than competition rather than waiting for a few months or years to have a more stable product.

However it probably can hurt a brand reputation in the long run if you have a quality level below customer expectation.

Customers are so used to things not working at times though that the bar is not that high.

So the quality we see would seem on average in line with expectations. Especially if things work most of the time and just fail to live up to expectations some of the time.

I’m not sure it’s as simple as that. I mean, isn’t capitalism supposed to drive innovation?

It seems to me that OP places a high level of usefulness or value of software at lower levels of abstraction, and I don’t disagree—it’s akin to how any manufacturing business is tremendously more valuable than the variety of consumer-facing products that can be made from it—but the cost of entry into such a business domain, and actually succeeding to make a profit, is often high.

The top five global companies by market cap, i.e. the "most valuable" by market metric, are Apple, Microsoft, Saudi Aramco, Alphabet, and Amazon.

From that I conclude that higher abstraction and making consumer-facing products through vertical integration (Apple) is more valuable than straightforward manufacturing. As is the old scheme of owning land with a valuable resource under it. And starting your company's name with the letter A.

OK, but then that’s not really what I wanted to say (and perhaps I really should have avoided the word “value”). Sure, those companies are the most valuable at the moment in terms of profits, but they are subject to the whims of customer tastes, which do tend to change every 10 years or so, and so they could eventually fall. A company that is in the business of manufacturing computer hardware, however, is far less affected by changes in consumer tastes because they could just make any computer according to the form factor of the decade, and could last much longer than the companies that use its outputs as raw materials or components.

So then if a company’s products or services lasts over generations, isn’t that actually more useful than another company’s products which people only throw away after some time? If the longevity of production of a product or service is not proof of the universality of a need, what is?

In construction, or most other fields of "traditional" engineering, projects will take years from start to finish. That's completely normal.

In the world of software - especially for startups - it is completely unacceptable to spend 5 years on building a MVP. There are some protected sectors where things move slower (defense, aerospace, for example), but if you're in the consumer field, you just can't spend too much time. You want to push out a MVP as soon as possible, and build on that.

And most software companies do not get penalized on shipping buggy software. Big game studios ship broken games, and spend a couple of years patching them up to their final form. People bitch and moan, but still throw money at 'em.

If you don't want bloated, broken, slow, and unreliable software - use your pocketbook.

The madness is insisting that applications, instead of operating systems, enforce security. In the desktop era, the OS was designed to blindly trust applications to do the right thing, and everyone went along with it.

That was the seed of the madness.

Instead of supplying dialog boxes(open, save, etc) for applications to use and then open files directly, the OS could have supplied handles (capabilities) from those dialog boxes for the applications to use. This would have allowed the user interface to be almost identical, and only require a few lines of code in the applications to change, in exchange for an environment which was almost immune to confused/rogue programs.

Because expectations were so corrupted in the desktop days, the situation now is effectively hopeless.

Applications should NEVER be trusted, especially not by the operating system.

> Instead of supplying dialog boxes(open, save, etc) for applications to use and then open files directly, the OS could have supplied handles (capabilities) from those dialog boxes for the applications to use. […] the situation now is effectively hopeless.

Not hopeless. MacOS, for example, added entitlements that get you there. Applications cannot list random directories or open files at will.

If an application opens a file browser, the operating system shows you your file system. When you pick a file, the operating system gives the application the right to read that file.

See https://developer.apple.com/documentation/bundleresources/en...

This is a way in which web apps are better. The security of what a web access can access (in terms of other services online) is way more granular.

Sadly, the Chrome browser uses a dialog box to let you pick things to give access to, then promptly never, ever, lets you see the choice you just made ever again. You have to close the browser to make sure things are gone, or if you made a mistake.

It's interesting to see how very different all this is from the original DARPA papers looking at security, because those all presumed that applications would be maintained by the system administrator and it was users who should be treated with suspicion.

> Applications should NEVER be trusted, especially not by the operating system.

This was basically impossible to do in the early personal computer era, before memory protection was a thing. So MacOS, Windows pre-NT, AmigaOS etc were all built around the assumption of applications reading each other's memory.

>This was basically impossible to do in the early personal computer era, before memory protection was a thing. So MacOS, Windows pre-NT, AmigaOS etc were all built around the assumption of applications reading each other's memory.

Very true. The user had a different option for that, since DOS was basically a program loader, and only one thing ran at a time, they could boot into program A, and use it with disks Q,R,S and know that no other disks could be corrupted. Then later, boot into program B, for disks X,Y,Z. It was capability based security, where each diskette was a capability. (Before hard drives ruined that trust model)

Many possible reasons, including:

- no engineering standards in computer science, including especially in education of computer science

- some corporate finance views which see technology as a cost center rather than a business enabler

- some corporate strategies where marketing decides what is possible and when it must be ready (tomorrow. or yesterday.)

- computing and software development as a "fun", creative endeavor - as opposed to a rigorous, formal process

I'm sure there are dozens more reasons.

It's simple. Hardly anyone wants to pay the premium price for quality software. Software is not physical goods and its real value comes from the things it can do it does not matter how it can do them.

Pareto principle. The last 20% take most of the work. People are not willing to pay for it (or at least that's what sales/marketing believe), so the effort is not being spent. It's as simple as that.

I feel your pain, but you really aren’t gonna be able to change it.

As long as people are willing to pay for dross, people will continue to produce it. Just look at Hollywood, for a classic example.

For myself, I found a company that was interested in producing Quality, and spent most of my career, there. It had its trade-offs.

It certainly didn’t get me much respect from today’s software development community. If anything, it gets me scorn and outright disrespect.

I had to retire and work on my own (for free), in order to finally get to write software the way that I want. I doubt it would be considered commercially viable.

In any case, “doing it right” is in the eye of the beholder. The fact that I'm not writing all my native Apple software in Haskell, is "doing it wrong," to some folks. Don't even get me started on OOP or using UIKit/AppKit.

Cant' please everyone.

I’m best off, putting my values into practice, in my own context, and not being an old man, yelling at the sky.

But I really want to. It just won’t do any good.

JSON is used because it is much simpler to generate and parse in browsers than xml.

It is also "good enough" for most use cases. So it is reasonable to start out with JSON and only upgrade to XML for complex documents/strutures.

JSON is also on average much more human-readable than the average XML document. Sure, you can do awful things with JSON and generate beautiful XML, but for some reason all of the XML I've encountered in production systems is extremely convoluted.

At least you can place comments in an XML file. With JSON, if you are lucky the application developer was lazy enough to not implement a strict schema validator so you can insert dummy keys... but other than that you're out of luck.

Also a different take on the problem is so much software is unreliable from a data perspective. We trust these software providers with GBs of our data, yet they all have different backup strategies, are using different consistency and ACID guarantees, and are not at all experts at managing and backing up data.

If we could abstract away the storage so that it is user owned, or at least a "utility" like electrical power lines where everyone is playing by the same rules, we would be able to trust these platforms significantly more, and rest assured our data is safe even if the software isn't.

"Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure."

( Melvin E. Conway, https://en.wikipedia.org/wiki/Conway%27s_law )

Thus, we may conclude the world has some people that choose to be "useless and unreliable". The software part is just an artifact of this ideology. =)

> will produce a design whose structure is a copy of the organization's communication structure

This is the original Amazon memo and the rationale for microservices.

The bigger picture is that if you look at the totality of the software and hardware system, from app to browser to OS to hardware, you see that it replicates the company and market structure that produces it. Which is partly why the web is so chaotic. The web is a standards battleground over which companies fight.

"This is the original Amazon memo and the rationale for microservices"

Ah, but the customer interface and product layer continues to degenerate regardless of the mission statement. The public always gets what they asked for... especially when it is cheap, awful, and disposable. =)

Engineering is not always implementing the best solution, but implementing a solution that completely or partially solves the problem at hand. In engineering, the fact that the solution is suboptimal is immaterial compared to the outcome. It may have ramifications to future problems that are possible following that, but that is a separate engineering problem that can be considered separately and decided upon accordingly.

Transferring ideas and software discovery are difficult problems. Expecting everyone to know that tech x is superior to tech y and singularly pursue development of tech x is unreasonable.

Most people prefer work with a solution that you are familiar with or caters to a particular technology stack. JSON is a first-class citizen of Javascript, but can be secondary in other programming languages. Anything that provides the least cost or effort to resolving or minimizing the extent of the product at hand is all that is needed. There is almost never a case where point B must follow or supersede point A. Point A continues to solve the problems it was designed for or turned out to be useful for by chance: the technology can continue to be used where it is known and regarded.

You don't have to use software, frameworks, tools that don't meet your standards.

I find that the learning curve to get on top of most abstractions is greater than what is required to understand the foundations. Knowing and applying foundational components tends to yield more performant and less bloated solutions. Debugging is easier because you don't need to unravel layers of abstractions which don't quite align with the domain you are working with.

Most companies don't do real cost/income analysis on any work. Most internal work is just approved by a department leader/tech lead. So most companies tend to overhire and do menial busywork, like constant refactoring of robust and well-working code, to catch up with the latest and greatest frameworks. The latest offender is vite, offering minimal gains (faster dev builds) at a non-negligible dev cost (and source of new bugs).

There isn’t any external pressure to give people an incentive to do better. Until that arrives, we’re stuck in this crap software local minimum.

I’m hoping such pressure arrives in the form of legal or regulatory stuff, even if it chills the industry and slows/shrinks it. Until then we’re still in the computing equivalent of the auto industry when people died in mild fender benders that today are just mild annoyances.

My two cents is that you are optimizing for engineering excellence. A more cynical person would optimize for making as much money in their career as humanly possible, which is made easier by pitching new tech that seemingly "Solves all the problems that old stuff had". A non-techy business person will naturally want to win market, beat competition, and do things faster. If you push that really hard you get snake oil engineering and churn out garbage.

Also, I assume most people fall in the middle. Wants to make money but also wants to do a good job. It just turns out those with the purse strings have the incorrect assumptions about tech, and if you know anything about business people you know that a laundry list of facts and reasoning is not always a good way to convince some of them of what should be done. More than likely a majority would see an article about a company going big with technology X. Now they have it in their head they need that too.

The harsh truth of the matter is that in B2B software engineering, you can usually get away with providing a shitty product, then charging to fix the bugs you introduced. It's worse than 'clients aren't willing to pay for a premium product', it's that they are willing pay you to clean up your own mess, but not willing to pay a much smaller sum upfront.

Internally at my small business we have stopped using the word bug, and started calling it defect to change the thinking about this. Bugs sound innocent, like they just 'creep in' and there is almost nothing you can do about it as a software engineer. Defects can be prevented, mitigated, or fixed. It places the responsibility in our hands again to deal with it and prevent it. Zero defects is probably not a realistic target (same with any production line for hardware), but it does help to frame it in the same way and take proper responsibility.

The reason is complexity, not necessarily that the companies don't care. Most end-consumer software is highly complex, using a vast number of dependencies, programming has become similar to LEGO brick building. Libraries are written by normal programmers, too, and even carefully designed libraries contain errors. Since the interfacing between libraries also produces bugs, the end product will become buggy simply as a matter of combinatorics.

The only way to avoid this complexity is by keeping programs simple or by writing everything in house in a language like Ada/Spark very carefully, including formal verification and extensive testing. This is way too expensive for non-critical software. Btw, I don't think Erlang's concepts are general solutions, fault-tolerance by restarting services only eliminates certain kinds of problems and isn't suitable for all high-integrity demands.

I don't think complexity explains it. There are plenty of industries that handle equal or greater complexity with greater reliability. I think the issue is that the industry is extremely cost-sensitive, and anything that lowers the cost of production is valued above quality. Not just by the industry, but by the industry's customers.

You see the same thing in a lot of places. When people talk about shopping at Amazon, Wal-Mart, etc., they don't talk nearly as much about quality as about getting the lowest price. Well, the lowest price means cheap junk.

Complexity explains it because the parts are not certified. In real engineering, parts tend to have precise specifications and sometimes must adhere to strict certification and testing. Outside of specialized areas, this is not generally the case for software libraries. If you accumulate such software libraries you cannot get error or failure estimates/models, and the result will be buggy no matter how careful you program or test or which programming language you use. The only remedy is to reduce complexity and/or develop everything in-house according to stricter rules.

Because it's not hard engineering where you can't rebuild the shitty foundations after you've poured the concrete. You can rebuild those multiple times a day in software, hence it's much more iterative.

I question what's your age and background, this view of the world is something I held when I was pretty young and thought everyone else were just stupid muggles. Until reality hits you in the face and you start to comprehend the constraints of the real world, software isn't some pure intellectual pursuit; it's a tool, a very very powerful tool that's used to enhance the real world somehow.

You can play around with pureness of theory if you want in academia. In the real world there are customers, they have needs and they want their needs fulfilled by a business, the business that achieves that earns profits, that's all there is to it.

When I think of bad software I think of software that works but has bad or hostile user experience. This is harder to solve than just learning linear logic and dependent types.

Ironically OP has answered his own question. CS is user hostile. There's far too much interest in tech for its own sake than in designing humane experiences.

In fact most commercial apps - banks, flight bookers, and so on - are usable. Not superb, not outstanding, but certainly good enough for the job. But the user friction comes from dark patterns and weird decisions.

Example: we booked a hotel on booking.com recently and there's an option to pick a double bed or two singles. Turns out that whatever option you choose the hotel gets a message that says "This booking needs a double or a twin room" and doesn't specify which.

There's probably a good reason for this bizarre and unreliable behaviours, but I have no idea what it might be.

The unreliability happens in the giant apps. Chrome had a problem with Streetview recently, and then it had a problem opening PDFs. OS updates are notorious across all platforms.

But these are huge, frankenstein projects with significant hardware dependencies. Even if you modularise them, formalise them, and test them, how are you going to do that for every possible feature on every possible hardware combination?

Making a useless and unreliable (say) toaster takes a lot of time and money. People notice fast that it's crap. Very few people are overly impressed by toaster styling and "features", or clueless about what a toaster is supposed to accomplish, or quietly intimidated by toasters, to overlook the fact that a crap toaster is crap, or quietly blame themselves.

Vs. with software, ~any clueless wanna-be kid can make some in an afternoon, at trivial cost. A PHB who could never manage to get a toaster designed & manufactured (let alone sold) can "lead" a team of crappy developers in creating some {cough} great {cough} new pieces of software.

The adoption curve of advanced technologies that solve it all. Is it just the cognitive burden of the tooling or the concepts, are the training methods different, a dearth of already trained talent and sufficient post-graduate or post-doctoral instructors, unclear signals between demand and a supply funnel?

The limited availability of Formal Methods and Formal Verification training.

The growing demand for safety critical software for things that are heavy and that move fast and that fly over our heads and our homes.

In order to train, you need to present URLs: CreativeWork(s) in an outline (a tree graph), identify competencies (typically according to existing curricula) and test for comprehension, and what else is necessary to boost retention for and of identified competencies?

There are many online courses, but so few on Computer Science Education. You can teach with autograded Jupyter notebooks and your own edX instance all hosted in OCI containers in your own Kubernetes cloud, for example. Containers and Ansible Playbooks help minimize cost and variance in that part of the learning stack.

We should all learn Coq and TLA+ and Lean, especially. What resources and traversal do you recommend for these possibly indeed core competencies? For which domains are no-code tools safe?

If we were to instead have our LLM (,ChatGPT,Codex,) filter expression trees in order to autocomplete from only Formally Verified code with associated Automated Tests, and e.g. Lean Mathlib, how would our output differ from that of an LLM training on code that may or may not have any tests?

Could that also implement POSIX and which other interfaces, please?

Not everybody should learn TLA+. It's a pretty niche tool and it's not helpful for a lot of kinds of software!

What are some of the limits of TLA+ for dynamic analysis? Which other tools model variable-latency distributed systems?

I don't think that most software is a variable-latency distributed system. I wouldn't use it for a simulation, for example, or anything which depends at a high level on non-integer numbers (TLA+ can't model-check floats or reals).

> The adoption curve of advanced technologies that solve it all. Is it just the cognitive burden of the tooling or the concepts, are the training methods different, a dearth of already trained talent and sufficient post-graduate or post-doctoral instructors, unclear signals between demand and a supply funnel?

> The limited availability of Formal Methods and Formal Verification training.*

From "Are software engineering “best practices” just developer preferences?" https://news.ycombinator.com/item?id=28709239 :

>>> From "Why Don't People Use Formal Methods?" https://news.ycombinator.com/item?id=18965964 :

>>>> Which universities teach formal methods?

>>>> - q=formal+verification https://www.class-central.com/search?q=formal+verification

>>>> - q=formal+methods https://www.class-central.com/search?q=formal+methods

>>>> Is formal verification a required course or curriculum competency for any Computer Science or Software Engineering / Computer Engineering degree programs? https://news.ycombinator.com/item?id=28513922

Formal methods should be required course or curriculum competency for various Computer Science and Software Engineering credentials.

> The growing demand for safety critical software for things that are heavy and that move fast and that fly over our heads and our homes.

"The Case of the Killer Robot"

> In order to train, you need to present URLs: CreativeWork(s) in an outline (a tree graph), identify competencies (typically according to existing curricula) and test for comprehension, and what else is necessary to boost retention for and of identified competencies?

A Multi-track "ThingSequence" which covers; And there are URIs for the competencies and exercises that cover.

> There are many online courses, but so few on Computer Science Education. You can teach with autograded Jupyter notebooks and your own edX instance all hosted in OCI containers in your own Kubernetes cloud, for example. Containers and Ansible Playbooks help minimize cost and variance in that part of the learning stack.

Automation with tooling is necessary to efficiently compete. In order to support credentialing workflows that qualify point-in-time performance, it is advisable to adopt tools that support automation of grading, for example.

> We should all learn Coq and TLA+ and Lean, especially. What resources and traversal do you recommend for these possibly indeed core competencies?

Links above. When should Formal Methods be introduced in a post-secondary or undergraduate program? Is there any reason that a curriculum can't go from math proofs, to logical proofs, to logical argumentation?

> For which domains are no-code tools safe?

Unfortunately, no-code tools in even low-risk applications can be critical vulnerabilities if persons do not understand their limitations. For example, "we used to have already-printed [offline] forms on hand [before the new computerized workflows [enabled by no-code, no review development workflows]]".

Should there be 2 or 3 redundant processes with an additional component that discards low-level outputs if there is no consensus? Is that plus No-code tools safe?

> If we were to instead have our LLM (,ChatGPT,Codex,) filter expression trees in order to autocomplete from only Formally Verified code with associated Automated Tests, and e.g. Lean Mathlib, how would our output differ from that of an LLM training on code that may or may not have any tests?

You should not be autocompleting from a training corpus of code without tests.

> Could that also implement POSIX and which other interfaces, please?

Eventually, AI will produce an OS kernel that is interface compatible with what we need to run existing code.

How can experts assess whether there has been sufficient review of an [AI-generated] alternative interface implementation?

There are a lot of very good answers already in this topic:

- quality doesn't bring enough to the business value;

- high demand requires hiring people without enough knowledge/skill;

- other people have just different quality metrics than you;

- "know-how" transfer between generations is difficult...

IMHO the similar question may be asked about any domain of life: why not all the buildings are well-built? Why the political/social systems are not just? Why good art is so hard to spot nowadays? They have different answers, but there is one common theme: building "well" is difficult; creating a "just" political system is "difficult", creating a good art is difficult; also writing "useful and reliable" software is difficult.

It's only natural that not everybody is a perfect software engineer, therefore is not a surprise that not every software will be perfect. What's more, there are a lot more mediocre software engineers than the perfect ones, therefore the majority of software will be... mediocre. The same about art nowadays — the sheer amount of music produced (even the word "produce" suggests it) every day makes sure that, in overall, there is a lot more terrible music compared to good compositions. I believe that software engineering has the same issue — the rising number of software projects is an enough explanation for low quality of the software we are able to perceive.

In fact, I don't believe the quality of the software engineering degraded over the years, but it's just more difficult to spot the great projects in the avalanche of "normal" life.

> What is this madness? This doesn't sound like engineering.

To me this sounds exactly like engineering. You craft the tools / apps you want, you build frameworks around complicated concepts that simplify their understanding and usage, at the expense of losing some fundamentalities.

You don't need to know the history and all the evolution of technology to apply it. It is part of software science, for sure though. You can be good at software science and fundamental concepts, but this also does not imply you are a good software engineer.

Would software in general be better if mentality proposed to you would be standard? I doubt, because the learning curve to enter the industry and even begin to start doing something would be immense, you would need to study as much as doctors now do, and only after 10 years you would be able to be "trusted" with your work.

I do agree though that there is a bare minimum that one who calls themselves "Software Engineer" should know and understand, like OS fundamentals, basics of compiler theory, etc., etc., but it is not so restrictive as you suggest.

I blame the desktop era. From the commercial standpoint, robustness is as good as reliability. And in the desktop paradigm, all the systems are robust enough because when something fails, there is always a user. A human part of the system that restarts the operation, or reinstalls the software, or, if nothing else works, buys an update (the best outcome for the software provider).

Because the cost of an error in different systems is different, and it is disproportionately lower than the cost of formal verification in 99.999999999% of cases. Where verification is needed and rational, it has long been used.

Because the only thing in common between the development of intangible products (software) and the engineering of tangible products is the cost of implementation.

Alan Kay has done a few lectures about exactly this phenomenon: his talk 'Normal Considered Harmful' in particular is worth a look as it pretty much goes straight to the heart of the issue: https://www.youtube.com/watch?v=FvmTSpJU-Xc

Searching for any of the comparison examples will surface the reasons. Rather than rant about it, why not propose a theory that generalizes the observations?

Personally I would tend to agree, that using a sharper more capable tool is moving in the right direction. Where I disagree is that we can't all reach for the sharpest, most capable tools, as that would render many/most of us ineffective.

The next question is if only the people capable of using the sharpest, most capable tools can satisfy all production needs. That might be possible for core things such as energy production, but likely not possible for all things down to the mundane. Or to remove judgement/controversy of 'mundane', all things people are willing to pay for.

That is an excellent question! I have been asking that myself all my professional life and I can now make an educated guess: lack of planning and mediocre specification.

I've read some comments here that I agree with; in a lot of cases the goals in dev are set to implement a solution to a problem either as fast as possible, or without a limited scope.

"We need a software that can fly rockets" - Yeah, but where to?

I studied computer science twice. Both times I have been taught to implement corner cases instead of working on clearly defined projects with scope and specification. "Write a small program that does X" yeah, but the solution space is unlimited, then, maybe? Can you elaborate? Put a scope on it? I mean, can do, may I order compute time on the campus super-computer?

Crap software was there ever since, and it got worse ever since.

> instead of working on clearly defined projects with scope and specification

Yes, it's much easier once you get to that point. Getting there is the real work. Wrangling clarity into requirements from messy humans isn't amenable to all the nice formal methods, but it's a very critical part of the work.

Could it be that formal methods are exactly the solution to "wrangling clarity into requirements from messy humans"? Business people and programmers could work closely together to incrementally build a list of formal requirements with the help of proof assistants. So that both parties can know clearly what are known to be ambiguous and what can be implemented immediately at every stage of discussion. Update the formal list of requirements whenever anyone has a breakthrough. Investigate into possible inconsistency early on with the help of proof assistants. Enforce project-wide constraints for every update to the list of requirements. Cooperate to make more and more informal stuff formal to reduce misunderstanding.

Formal methods can help with exploration and cooperation.

"Business people" and programmers are inherently incompatible, because there is a clash of interests. What you need in dev team is a programmer who balances possible solutions against requirements, and only their decision counts, and guides the other programmers. "Heya, our software is gonna be great if we add this nifty feature I wrote last week" then the deciding has got to say "save that for the next major release, let's get the requirements working bug-free and testable first!"

Nobody cares.

Customers will not pay for quality. Not in toasters, not in cars, and not in software. Cheap and fast wins the day.

Programmers are driven by the need to get stuff done and they migrate towards technologies that do that quick and that are sufficiently documented.

You want your "fundamentals" to be learned? Make knowing them useful to achieve something and document them in nice skimmable, searchable manner.

One pattern I noticed in the workplace: People are mocking things like LISP, Haskell, or Erlang, but get a hard-on when their bread language (e.g. Python) introduces new syntax for, say, pattern matching (rather a pattern-aware switch/case, because it isn't an expression).

Software is dysfunctional because everything is dysfunctional. Any human endeavor done at scale is going to be a complete clusterfuck of a trainwreck. It's just more obvious with software because we're closer to it.

Civilization is a goddamn miracle.

I believe software has the potential to be an exception of "any human endeavor done at scale is going to be a complete clusterfuck of a trainwreck." In software development, there are methods to scale correctness not mistakes, but they are not as prevalently used as desired. Some practical examples being:

1. Affine types in Rust.

2. Refinement types in Ada.

3. Dependent types in Coq/Agda/Lean/Idris/ATS-lang.

4. Separation logic in Facebook Infer.

These methods systematically/automatically eradicate errors that incomprehensive/human-intensive alternatives (unit-testing, code review) can't do at scale. If our industry could adopt and improve on this, we could create truly ambitious projects with confidence, at a lower cost, and at a faster pace because 1) correctness is scaled and 2) mistakes are contained.

> What is this madness? This doesn't sound like engineering.

Did you hear that everyone can build a bridge that stands but it takes an engineer to build a bridge that barely stands?

I'd say software development has a lot to do with engineering then.

In case of civil engineering, there is huge amount of regulations.

We have compilers.

> Protobuf, flatbuffer, capnproto, and the like keep reinventing ASN.1.

This is a strange argument in the context of your other complaints (which I largely agree with fwiw). All of them are much safer to parse than ASN.1, which has been a decades long security nightmare, and add additional security concepts on top.

Never mind that they're also networked IPC protocols on top of binary serialisation formats, solving a host of security issues we've had with exchanging serialized Java/Python/PHP objects over the wire, DCOM, etc. pp.

Damned if you do, damned if you don't?

Craftsmanship is not valued the same way by everyone, and it costs a lot more. The people paying for software (either in salary or in license fees) have to choose their battle: quick and cheap is generally the default target.

Most likely, you also chose cheap over quality at some point in your life. I have furniture that would make a woodworker cry, sportswear that raise eyebrows, unreliable WiFi and bad cables. How can I judge people that don't even know what quality software is?

Everything is crap because people can't afford to have the pinnacle of human creation for every product in their lives. The crap one will suffice just fine.

Like any other product, software has different levels of quality to it.

Most people are not prepared to pay for that level of excellence. They want the cheapest thing as quickly as possible.

The only way that this will improve is if the "baseline" quality of software improves so the average coder automatically produces better stuff because the popular tool of the day is providing it for "free" without them really thinking about it or having to opt-in to it.

Because of "Agile" planning practices. Get things barely working (MVP) then move onto the next project to make sure your manager's manager stays happy.

It does seem like software quality declined as these methods rose in adoption. But most software was crap before that, too.

> Naive microservices partially reimplements Erlang's BEAM VM while ignoring all the hard parts that BEAM VM got right

Can someone elaborate or give hints more about this?

Because it is funnier to write new shiny things than boring, reliable things.

I have an education in computer science, I know how to write a Paxos setup, I am fluent in Erlang. I am still very much not interested in formal systems, proofs and verified computing, because there I will spend 10x much time for the same set of features. Bugs are normal. I will live with bugs. I won't live with 10x development time lost.

So lack of regulations then?

Welcome to normal life. Almost everything is crap. Almost nothing is like a high quality event, if you mean. There's a thing that maybe you have heard about called Normal distribution. You can only call something high quality because a little few extraordinary things happen. Otherwise it would all just be ordinary.

Welcome to normal life. Almost everything is crap. Almost nothing quite like a high-end event, if you will. There's something you may have heard of called the normal distribution. You can only call something high quality because some extraordinary things happen. Otherwise it would all just be ordinary

It all boils down to capitalism. Capitalism won't allow you to spend time and money building quality, when someone else can do the same job twice as fast and for half the time. Getting first to marked is more important for most than having software that work all of the time.

The only reason that other engineering professions does things properly is that bridge building is not free marked capitalism, but government mandated rules. Just look at how much GDPR rules in EU does for actually making companies do the bare minimum to respect privacy.

Unlike most fields, anyone with a computer can program

Ivory tower academics will keep insisting that there are ways to make foolproof software.

If it were so, they could do that and make millions and trillions.

Software is not a science, its a craft, it depends on the outside world, not some cute lab demos

Anyone with a education can write. Almost everything that is written is crap. There's no literature genius saying that normal people should not write or even should write with a rulebook aside.

> If it were so, they could do that and make millions and trillions.

Not disagreeing with your fundamental point, but it's also true that most people don't value making huge piles of money as much as other goals.

Because people pay for it.

People don't pay for incorporating full functionality of "XML...Whatever else" they pay for ability to see cute kittens NOW not when dev will be done with reading and understanding ASN.1 well enough to use important parts.

1. There are much less really good programmers than people who think they are ones.

2. Business dont care much about quality, as long as the product is good enough to sell - real improvement and development stops there.

To some extent I agree - I do see a vein of anti-intellectualism in software. But “engineering” isn’t about using the most theoretically-perfect setup for any given job, it’s about navigating practical trade-offs.

The software in a typical enterprise has to cross a lot more domains then it used to, and has to work with independent organizations over the internet. No surprise then that it has more confounding elements.

There was a great white paper investigated how software gets bigger and bigger. They used the command line tool true as an example. I can't find it anymore but it was really an eye opener.

There is a lot more software now than there was in 1987. The percentage of bad software could be the same. Sturgeon's law etc...

Because everyone wants to optimize for highest incoming cash with the least amount of time spent? :)

the world runs on compromise and is powered by hope of perfection.

For the same reason you're posting snark and not a Show HN.

Sturgeon's Law.

Precise. In my country we say that you hit a fly's butt.

> This doesn't sound like engineering.

Because it isn't.

Economics.

Reply on news.ycombinator.com