How Software Gets Bloated: From Telephony to Bitcoin

1. The user/client comes across edge cases, and has a very specific thing they want done in that situation. Unfortunately, it's the complete opposite of what the system would usually do, so said case has to be specifically patched to act in a certain way despite making the codebase a confusing mess to figure out. So one tiny part of a certain web page gets different padding or line height because the client/user thought it looked better in that one situation, and no one thought to say it was a bad idea.

2. The project's spec was wrong and hence it was the wrong thing to be building in the first place, but no one liked the idea of starting from scratch. So you end up with a program that's meant to do one thing but has been crudely hacked into doing something completely different and hence half the features are utterly useless.

So it's not the developers fault in these cases, its management not knowing what the team is actually meant to be building or spending a lot of time trying to provide exceedingly random and useless 'nice to have' touches.

From a marketing perspective it is easy to see that "Adding Feature A will help us sell to customer B." It is not clear (and probably not true) that feature A will cause us to lose a sale to customer C, but the cumulative effect of a huge number of features can lead to "bloat" and "unreliability" that can cost sales to customers Y and Z and drive the cost and risk of development so high that it stops.

I think of Adobe PDF as an example. People criticize it as being "closed", which is not true, it is one of the few ISO standards which you can download for free and Adobe even scrupulously documents how their products deviate from the standard. However, it is insane how much stuff has been added to it beyond a document format (i.e. exchange of 3D models) such that when I am trying to control the size of the C: of machines I control, Acrobat is the Creative Cloud product that is the most universally useful AND one of the absolutely largest thus making it the hardest decision.

The arguments against it being "closed" and "bloated" then crowd out the real criticism you'd have if you actually read the ISO standard, such as the fact that you can't draw a circle in PDF. (I don't know what happens if you 3d print a wheel made out of 4 bezier curves that look like a circle, but I am sure some of you will find out.)

Thanks to Wikipedia, I found that this may have been an earlier one-liner by Greg Kuperberg:

Every program in development at MIT expands until it can read mail.

http://groups.google.com/group/rec.humor.funny/browse_thread...

But sometimes constant employment on a project just creates bloat, too. Refactorings that are intended from the beginning to reduce complexity either don't end up delivering, or they are never completed, leaving systems in a half-migrated state. Most systems I am familiar with are in a perpetual state of transition between the old way and the new way. The more complex the system, the harder a migration is. The harder a migration is, the more likely it is not to be fully done. The more incomplete migrations, the more complexity. On and on.

It's why eventually we should keep trying to reboot software systems with from-scratch, feature-lean replacements.

Software that is not used (for what ever reason) is abandoned, and therefore does not grow.

So there is selection bias too - the most notable software is the successful cases, and they become "bloated" because they are successful.

(what the author talks about is not bloat, it's clever hacks that also make it much harder to understand the program)

As a developer, I frequently categorize and define bloat as large amounts of extraneous code from libraries and other tools that are not directly used by any part of a given application, but are nonetheless imported or installed by a project. Here, I typically go pointing the finger at requirements files, sloppy imports, and other dependency-related smells.

As a manager, I may define bloat as anything in the above dependency realm when I'm working with developers. Alternatively, anything in the realm of features and functionality that do not have a defensible position as being built for and used by a majority of users is sure to get some dirty and suspicious looks from me as bloat. This is typically triggered when a client or other project stakeholder suggests Feature X be added so Person Y (or Customer Z) doesn't have to use some other Tool B. I push back hard on such suggestions until there's a demonstrative case to be made that this kind of bloat won't become a major liability in the code from a long view, or a stumbling block for the majority of users.

As a user, I often define bloat as any number of features I don't personally use and, through various interactions and use-cases, feel get in my way, clutter up my UI, whatever.

Everyone involved (except users) is responsible for the bloat in their own domain.

Bloat comes from the combination of endless feature requests combined with the need for quick development cycles to implement those features.

Developer competence only slows the process.

Bloat is not the same as complexity, but they are related. Bloat is not always bad, or even perceivable as such by the consumer of the code, but at the same time it's also never good. Bloat is a natural accumulation tradeoff decisions.

Anybody who writes code produces bloat, because writing software is an exploratory activity towards solving a problem set. There is no way to always know where the bottlenecks are going to be, and where your chosen abstractions will break down. The difference between doing good work and bad is that the programmer doing good work will revisit decisions and refactor things ruthlessly down the line. The programmer doing bad work simply doesn't care, is too pressed for time, or writes code in a way that makes refactoring prohibitive.

Personally, I think a good litmus test for bloat is how much the code you're writing actually achieves for the effort you put into it. If you find yourself constantly massaging and circumventing the limitations of your boundary layers, that's a big warning sign. If you are using a library or framework that prolongs and complicates the effort of solving your problem compared to the expected effort of doing it from scratch, then that library is clearly not useful. Using it regardlessly is a deficiency in the critical thinking required to make good decisions in programming.

I am quite happy with software written by people who who do not view themselves as "devs" who are "smart", "the smartest", or the "most technically competent".

I'm also happy with software written by telephone company employees in the 1970's, a grad student in the 1980's and a university maths teacher in the 1990's. Software written by "security researchers" is usually not bad either.

I like the small programs that win the IOCC; if I were asked to name a test for "competence" (my definition), the IOCC would be high on my list.

I have no reason to question that the "devs" writing the bloat, e.g., at Microsoft in the 2000's or any number of companies today, are _brilliant_.

However I have little interest in that class of software. I think bloat is stupid. Not to mention unnecessary for my purposes. If I cannot pick the code apart and recompile it myself, then the software is nothing but a liability to me, not an asset.

read thru it... think we could rewrite it smaller and capture all the wisdom and make it more orderly? would you care to join the team and start doing that? :)

The road to hell is paved with good intentions....

Sometimes when we add a requirement, numerous complications cause new lines of code to be added throughout the program. So the requirement doesn't have a fixed cost in terms of additional size; the bigger the program already is, the more lines are added by the new requirement.

Someone wants rational numbers in the numeric tower. Okay, so we have to implement addition of a rational to every existing type, including itself. Multiplication of a rational by every existing type as well as itself. Subtraction, division, ad nauseum. The cost in terms of code size for the requirement to have rationals depends on how many numeric types we already have.

Development by accretion of third-party code is the major cause of bloat. Third party components are already getting bloated over time due to the global bloat effect. And here you are, just mashing these components together to make your program. You don't use anywhere near the full functionality of these components. You link that entire five megabyte shared library because it has three useful functions. The seventy-seven others you don't use have to stay there because it's a shared library; some other program might attach to it and use a different subset.

These components get upgraded over time. Security fixes, enhancements, loss of support for old versions. Several years later, you're still just using only three useful functions in that shared library, and none of them have changed. But it's no longer five megabytes; it's now ten and has 157 other functions that you're not using, not 77. You hadn't even done any work to get that five more megabytes of bloat; you just triggered some automatic update a bunch of times over several years and ran some regression tests that passed.

Even the same code gets bigger when compiled. Thirty years ago, the functions were packed together, as well as their internal basic blocks: there was no advantage to any special alignment for branch targets. Loops were not unrolled and code wasn't inlined much: caches weren't big enough so it was often counterproductive. C++ templates came into the picture since then: let's generate umpteen variants of the identical code into slightly different machine language dependent on type.

Managed languages: these drag entire platforms with them. And the internals are "not user serviceable": you generally don't rebuild these things to remove what you don't want. You accept the platform with all its bells and whistles and work your app into that. Their bloat is nothing new. In the 1980's, some Lisp implementations were criticized for making 20 megabyte images (at a time when that really hurt). The rest of the world caught up to that: we now have non-Lisp languages that have lexical closures, garbage collection and ... huge images, far in excess of 20 megabytes. Today's managed languages, being platforms, have naturally copied the concept of package management from actual operating system platforms. It's super easy to bloat up the image just essentially by adding simple declarations of what you want in it, chosen from a large online catalog.

That's not my experience.

My experience is that software bloat comes from people who think they know what they're doing, but actually don't.

Competent programmers tend to be much more cynical about their own ability to write good code. They've been burned too many times by bad code, including their own.

The cases discussed here are not about competence or brains. They're about upgrading systems, which maintaining compatibility with existing systems. That is very, very, hard for anyone to do.

In other cases we also end up spaghetti code software full of hacks and bad designs because it wasn't well thought out or designed from the start.

In the end designing software that matches expectations is the biggest culprit of 'bloatware' vs 'spaghetti monster' codebases.