Hacker News Reader

Comment by hcarvalhoalves | Hacker News Reader

It's the Swiss cheese paradox: "cheese has holes; bigger cheese have more holes; therefore, the more cheese, less cheese".

> However, I still haven’t found any studies which show what this relationship is like. Does the number of bugs grow linearly with code size? Sub-linearly? Super-linearly? My gut feeling still says “sub-linear”.

We know from observing reality that even buggy software is better than no software – a software so buggy it adds negative value is a rare exception. So I guess it has to be "sub-linear", otherwise software wouldn't have economies of scale.

The contradiction though is: how do we explain the apparent stability of large but mature codebases? Thinking something like Emacs for example here, where some commits date back to 197X. I guess at some point it becomes an inverted U shape even if codebase grows, if it grows at a conservative pace.

View on HN

> a software so buggy it adds negative value is a rare exception.

Survivorship bias will explain all of that. You'll rarely encounter negative software because it gets discarded. It doesn't mean it's never created, it just dies before getting wide visibility.

> otherwise software wouldn't have economies of scale.

The fact that software can be developed once and sold indefinitely means the marginal cost is near zero. That is an unbelievably huge economy of scale that would dwarf many other superlinear problems.

> how do we explain the apparent stability of large but mature codebases?

Not all software changes are additions. If assume that codebase size is positively correlated with number of bugs (i.e. more code = more bugs) and that most changes that do not grow the codebase are likely to fix at least one bug, then mature codebases should be expected to improve in quality to the degree that number of changes outpaces total codebase size.

I had to kill one of those projects. It was so buggy it was causing issues in "unrelated" areas. I find out Thanksgiving week if I killed it early enough for the next release.

> The fact that software can be developed once and sold indefinitely means the marginal cost is near zero. That is an unbelievably huge economy of scale that would dwarf many other superlinear problems.

I'd propose a slight revision: "software can be developed once and used indefinitely." I would bet that most code nowdays is written inside organizations and never directly "sold" over and over again to a customer. But even that tends to result in those organizations wanting more and more, because even a buggy one tends to save people time compared to doing the process manually, and then there's another process they want to automate next, etc...

I'd propose replacing "indefinitely" with "until the systems it interacts with require changing it". It takes surprisingly short time for a mobile app that is written to current standards to turn into something that doesn't run on latest updated OS version. It's a good point that selling software indefinitely isn't possible. There are only a limited amount of people who want the software and are willing to pay for it.

It's still effectively "sold" in economic terms. You have one codebase and it generates revenue per customer (where "customer" may either be an end user, or an advertiser for ad-supported software).

Software size is positively correlated with bugs, but software age is negatively correlated? If your codebase doesn’t grow much in size but grows older, the quality should become better.

The best software is very old and has been polished to a mirror-like gloss.

Having any ongoing development means you get an equilibrium level of defects, though.

> The contradiction though is: how do we explain the apparent stability of large but mature codebases?

The problem with these sort of analyses, in my mind, is that they seem to neglect the iceberg of code you build on top of, which works perfectly fine and is virtually bug-free. For instance, imagine that I'm building a website. If size is a predictor of quality, then you'd expect me to add bugs as time goes on, and that's probably true. But my site is stacked on top of Chrome, an abstraction with hundreds (?) of millions of lines of code that virtually never give me an issue. This is stacked on top of the typescript compiler, the C++ compiler, the operating system, etc, and those all run in the millions (TS) to tens or hundreds of millions (OS) of lines of code! So while I don't doubt that codebases get buggier over time, this seems to neglect the fact that I use truly immense amounts of code every day that don't seem to run into an issue. Perhaps the bugs grow around the edges rather than in the center, or something like that?

I would guess that this theory applies average codebases of average complexity and quality.

Chrome, popular compilers, popular operating systems skew towards the highest end of quality and developmental rigor.

Chromium has 63,438 open bugs in its bug tracker so the theory remains true, but maybe the broader point is that strong abstractions have enabled us to reach great levels of complexity. Testing, staffing, and programming best practices probably go a long way in making software like Chrome appear to work flawlessly from the outside.

An analogy would be the craftsman-factory. A craftsman knows and performs every step of the process, and the end product depends on the skill of the craftsman. While a factory each worker is limited to a specific job and the combined effort of factory workers allows products well beyond the reach of any craftsman.

> Chromium has 63,438 open bugs in its bug tracker so the theory remains true

How many of those bugs affect you on a day-to-day basis? For me, the answer is 0. In fact, I can only think of one time ever when I was affected by a browser bug as a developer. I think that most of the reason Chrome has so many bugs is probably just because a staggering amount of people use Chrome, meaning they can inspect every nook and cranny.

For a somewhat contrived example, imagine I write a program to add two numbers together. I write a single line of code: `var foo = bar + baz;` in JavaScript. If I ship it to no one, I can claim it has no bugs. But if I ship it to a million people, then I might start getting bug reports from people running it under extremely precise circumstances and running into issues. For instance, maybe I'd get a bug where two numbers sum to be negative because the user ran into integer overflow when bar and baz are very close to INT_MAX. I might also get reports of lack of precision when bar is extremely large and baz is not (e.g. sometimes foo + 1 === foo in JS if a is huge). I suspect most bugs in Chrome are of this variety, or, more likely, they are even more obscure.

It seems to me that it's not a strict 1:1 correlation between size and quality - there's also a factor where more popular software has a lot more people looking for bugs.

> e.g. sometimes foo + 1 === foo in JS if a is huge

Superfluous use of the strict equality check there. An ordinary comparison (i.e. using `==` aka "double equals") is sufficient.

How do you know that foo isn't a numerical string?

Because I'm responding to the scenario you actually laid out, not counterfactuals.

https://bugs.chromium.org/p/chromium/issues/list

The majority of the bugs I see are failed tests, null dereferences, unreachable code reached, very slow code, UI bugs, crashes, etc.

Come to think of it, I've had 2 chrome bugs on my phone in the last week or so. Some weird UI bug with tabs, and a more severe one where it just kept crashing once in like 10 seconds. It magically stopped after a while.

Things like chrome, or Linux, are so complex, and interact with so many different variables that it's completely understandable that it can have tens of thousands of bugs, and yet most people won't ever notice one.

Maybe another variable at play here is:

Evolution: modifications_size / time / codebase_size

Mature systems tend to have lower Evolution. Thus, there are more bugs being fixed than created. Over time, this leads to stability.

On the other hand, one bug fix creates three more bugs, on average. So, I don't know why on Earth stable systems are even possible...

>one bug fix creates three more bugs, on average

Sounds really interesting, where is this stat from?

It contradicts the stat that I have heard (which I can't document either): Fixing a bug has 20%-50% chance of creating a new one.

Just kidding! :D

> why on Earth stable systems are even possible...

They aren't, but it just looks like they exist. The more complex something is, the less likely it is to notice a bug in the complex dependencies.

Taken literally - "why on Earth", as in, in nature - it becomes a matter of timescale. Something we seem to be only fully realizing now - ecosystems and climate are in constant flux; they may look stable on a timescale of a human lifetime, or the lifetime of a society, but at the scale of several centuries or more, they're chaotic.

> Mature systems tend to have lower Evolution. Thus, there are more bugs being fixed than created.

Ideally. But in practice, they must add features to sell more copies. And even if they have no evolution, OS's evolve that make the software worse over time (e.g. "modern" UIs, hardware vs software rendering, etc.).

We see this in many products that were once best-of-class, e.g. Paint Shop Pro, ACID Pro, etc.

It's not so much "sub-linear" as "capable of being bypassed".

The CEAC State Department website is the most common one for me... the file upload only works in incognito and "once" per session. So people have figured out that you can log out, log back in and keep uploading your immigration paperwork.

The larger the piece of software, the higher the chance of a workaround for any given bug existing?

> We know from observing reality that even buggy software is better than no software – a software so buggy it adds negative value is a rare exception. So I guess it has to be "sub-linear", otherwise software wouldn't have economies of scale.

Your logic is flawed here. First, software has economies of scale because it's easy to produce the second copy of a program once you have the first one. It has nothing to do with lines of code.

Second, buggy software being better than no software would still be true even if bugs were super-linear in the number of lines of code.

I'm pretty sure that bugs aren't sub-linear. As you pile on more and more lines of code, the number of bugs per thousand lines goes down? The code gets better as you create more and more of it? Experientially, no.

I think it's super-linear, for a couple of reasons. First, the bigger the program, the bigger the team. The bigger the team, the more below-average coders on it. Second, the bigger the program, the more room for weird interactions between obscure conditions.

No, it's definitely not sub-linear. Bugs per line of code do not decrease as you have more lines. I suspect it's

A bigger team doesn't create buggy software because it contains below average coders.

It creates buggy software because the team members have to coordinate with more people.

> A bigger team doesn't create buggy software because it contains below average coders.

But it must, by the very meaning of "below average coders" - it's implied that we're rating hypothetical coders on software development skills, which include the ability to make correct decisions (both at code and architecture level).

> It creates buggy software because the team members have to coordinate with more people.

Those are not independent variables. The bigger the team, the bigger coordination problems, the more bugs. But also, ceteris paribus, the worse the coder, the more problems there are coordinating with them, thus the more bugs. And also, ceteris paribus, the worse coders are on average in the team relative to global average, the more of them are needed for a project, which makes the team bigger and coordination more difficult. Etc.

Number of bugs can grow super-linearly while their impact might be decreasing preserving the overall usability increases. Basically as your software grows there's more corner cases that will get tested very rarely if ever.

Number of possible states grows almost exponentially with code size, so it would be very weird if the number of bugs grew sub-linearly.

> We know from observing reality that even buggy software is better than no software – a software so buggy it adds negative value is a rare exception.

Whom observing which reality?

I have made no scientific study - I am not stating any sort of strong opinion, just an uneasy feeling - that about 90% of the software and uses of computers subtract from humane well being and welfare, not contribute to it.

Looking at you Javascript frameworks.

Looking at you databases and tracking.

Looking at you spyware and advertising bots.

Looking at you killer robots (yes they exist - mostly thinking of the horrific USA military drone assassination programme)

So many examples to choose from. I should not complain, I make my living from computers and the uses I facilitate are no shining examples of improving people's lives

i think you are mixing two concepts s imperfect software like databases and JavaScript frameworks, the others aren't imperfect they are evil, they produce what they want to produce, yes is negative for society, but not incorrect.

> how do we explain large but mature codebases

If the codebase/project became "feature complete". Once a project becomes feature complete, then finally bugs are able to be squashed without reintroducing new ones. The code becomes better because the deaign begins to accurately model the program since there is no feature creep.

The problem with large programs is when you _continue_ to add on to them. Non-open source very rarely gets to the feature complete stage.

> how do we explain the apparent stability of large but mature codebases

I would hypothesize three potential factors are at play, especially for software like Emacs.

The first is that some bugs becomes, themselves, "features" over time. It might be wrong from the standpoint of what was intended or desired when the code was written, or what is documented, but because that is how it works, that's how it is now expected to work, and so it's no longer a bug.

Another factor is that some bugs are "shielded" by the surrounding code, effectively two pieces of code offsetting each other's bugginess. Unless and until you either replace one or the other, or need to interact with one or the other in isolation, the bug isn't apparent. Kind of like how going to the chiropractor for one problem can "unlock" other, latent, problems.

The last one, somewhat related to the first two, is that some well factored software systems do a very good job of isolating their most stable parts, leaving their less stable, more commonly changed or iterated parts in separate modules or components. Because the change rate and focus of effort is on the "outer" parts of the system, less time and energy is spent on the "inner" part. Very often, developers of the "outer" parts will accommodate the bugs of the "inner" part, because changing the inner part requires both greater knowledge of the overall system, as well as the potential for a larger "blast radius" (and therefore assumed responsibility) for ones change. So rather than debugging the "inner" part, the "outer" part grows lots of accommodations for the inner part. It's not until that inner part can no longer be accommodated, or it becomes a security vulnerability, that the community developing the system will see the value in "fixing" the "core".

Anyhow, just a set of hypotheses.

eMacs has the Church of eMacs backing it, so it might not be a good example of a normal codebase. I personally feel a compulsion to dedicate time to giving back to the eMacs ecosystem in a way that I can’t extend to any other codebase, and there are others that I likely depend on to a much greater extent.

Jokes aside and that being said, I agree that there is something helpful about slow, organic (in your words, conservative) growth. It’s neither necessary nor sufficient for stabilization, but it certainly seems to correlate. GNU in general seems to be a good example.

> otherwise software wouldn't have economies of scale

Does it? I tend to think it doesn't (in terms of code size; obviously it does if we're scaling by number of users)

Even if bugs scale sublinearly in practice- my gut tells me it's because development as a whole slows down even more than the bugs accelerate, partially making up for it

I.e. suppose your codebase size grows 5x, and the bugs per change grows 10x, but then the org's rate of change also slows down by 5x. Now your bugs per time has only grown by 2x.

Agree that a lot of it has to do with rate of growth. Compare emacs or postgres codebases with some of the horror shows that are the codebases at VC-hyper-growth startups.

If you fix commonly occurring bugs, you're left with rare bugs - and there may be tons of them in mature codebase. For legacy systems with dedicated operators, it may make more financial sense to teach users a workaround to a bug, than to fix it.

I'm not sure about this, our e-health system had millions spent on it and eventually still was discarded, after "sort of" working for months and slowing down all of the doctors and pharmacy workers who had to deal with it, as opposed to just using paper slips for prescription medicine, or even individual hospital data management systems: https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/...

In addition to instability, even in my country, Latvia, there have also been large enterprise systems that have had serious performance issues, like the government e-system platform latvija.lv went down when the COVID vaccination signups became available, which was problematic, given how many other systems depend on it as a login provider. We've also occasionally have had the national tax system go down in the dates when people can start submitting their tax reports.

In regards to security, you occasionally see all sorts of problems even in large enterprise codebases, a recent case was the Krete education system source code (not in my country, just a relevant example), where for some reason code like this was running in production: https://www.reddit.com/r/ProgrammerHumor/comments/yqp2o6/our... here's an article about it https://hungary.postsen.com/trends/86603/The-CRETE-study-sys... (though it got hacked in a phishing attack)

Also, there have been cases of problematic deployments bringing down entire orgs, Roblox had their core business be halted for a while a year ago because of problems with their Nomad cluster, which resulted in a 3 day long outage: https://roblox.fandom.com/wiki/2021_Roblox_outage

The Knight Capital Group also lost more than 400 million USD in less than an hour and went bankrupt because of a bad deployment: https://dougseven.com/2014/04/17/knightmare-a-devops-caution...

I've personally seen systems that have gotten so complex that they're a liability - if there is no full test coverage (as close to 100% as you can get) then you're usually under the risk of wrong calculations or behavior in the system, which may or may not be possible to roll back, depending on the nature of the system. Sometimes it might not even be easy to detect, but the implications can be pretty great, especially if you're dealing with finances. Or having people depend on the systems and not having a business continuity plan were they to go down, due to how integrated things are in our current society.

There is plenty of software out there and everything around that software (other code that determines how it will run, how it will scale, how it will be secured) that has plenty of potential for creating lots of negative value, all the way up to making organizations evaporate, or causing lots of suffering for numerous people. One doesn't even have to consider something like the software inside of planes or software that controls train routing or power distribution networks.

Reply on news.ycombinator.com