Hacker News Reader

> This does not happen with products that are highly successful business-wise, no matter how bad the suffering, so it's really a business failure rather than a technical one.

One thing often feeds on the other. Because the system is hard to change, it does not get necessary features. Because it does not have necessary features, it provides less business value. Because it provides less business value, there is less of a budget for improving it. And so on.

Bingo.

Some of the worst examples of that are when a project uses a custom build of a library of which the source no longer exists and no record of the changes exists either.

Reintegrating future changes in the upstream is also made nearly impossible as a result; our tech lead made a change to numpy a few years ago, didn't manage to get it accepted by the project, and we're stuck with this version until the sun burns out.

If there are changes in future numpy versions we want, it's up to us to backport them, which is nowhere near our core business.

There's a lot to be said for standardization and 'boring.'

Well you could estimate the impact of backing out of the changes on the application side, with the upside being continued savings in operational complexity. Or, you could address the operational complexity with processes or tooling - which would be easiest with something like a developer OS image.

Yep, seen that. You might be thinking of tweaked builds of open-source components, but before package managers were so common, I've seen also internal projects with a `/lib` folder full of artefacts like `MiscDbUtils.dll` which are internal "useful" utility functions that are widely used.

Now add in a script that updates this artefact with the latest version, breaking changes, and it all goes wrong and it's hard to find the correct previous version of the binary artefact to build your code any more. Especially if the build has been broken for a while due to the project being on the back-burner and it quietly dies when no-one is looking.

Builds that involve non-version-controlled files that exist only on a certain developer's machine, and because that developer is a control freak (or overworked), he refuses to automate or put those files in version control...

I believe this is called "Job security." I've seen it a few times including a dev deleting the source code repo and substituting all copy of a set of scripts he was responsible for with compiled binaries. This was discovered due to a platform incompatibility between one of the hosts running the script and the binary wrapping. Data was then restored from backups and dev was summarily let go.

I should add here that the "antiquation" case is the one that has caused the most observed grief in my career. The forces causing failure are coming from outside the code/business (dependence on an outside vendor), and sometimes collide with forward momentum of other parts of the code (i.e. that graph library will never, ever work with Java 7, to name an example). These become life-or-death situations, and the tendrils of the product dependency are often deeply integrated. It might be easier to rewrite than to fix.

Also, this case can impact not just products, but organizations. You can still find teams dependent on an antique commercial version control system or IDE that greatly slows down or even stops work. I've tech-led jumps to new version control systems a few times, and it's always riddled with anxiety, strain, and management angst. (And it always makes the team far happier and more productive!)

>You can still find teams dependent on an antique commercial version control system or IDE that greatly slows down or even stops work.

Sounds like Rational ClearCase.

Yeah, that. But even things like cvs, that were modern and hip in 1998, are still floating around 20 years later.

I actually thought Subversion would be the last version control system, when it came out. Of course, now it's git. Maybe someday we'll get something better, and git will look decrepit.

I have little doubt that git will be replaced eventually (or perhaps severely modified). It seems like a fad to me. It is indeed very powerful, but its UI is sheer insanity. At least SVN is very straightforward to use and understand. It just doesn't offer the distributed nature that git does, as it relies on a centralized server.

The cool thing about Git is that the underpinnings ultimately boil down to a key-value data store.

I'm not an expert on these innerworkings, but in theory there's nothing stopping someone from creating a new UI that maintains most or all of the same strengths, except that everyone already uses and is used to the current way.

I suspect if you came out with "SuperVCS" that was ultimately just a new UI on Git you'd have more success than releasing the exact same project as some kind of Git enhancement.

Isn't that basically Gitless? (http://gitless.com/)

What UI? Using git on the command line is exactly the same as using SVN on the command line. At least for basic, every day things like status, add, commit etc.

> I've tech-led jumps to new version control systems a few times, and it's always riddled with anxiety, strain, and management angst. (And it always makes the team far happier and more productive!)

In the cases of this I've seen, it's always been because management and team priorities were unaligned.

Management in those places cared about a minimum level of productivity and minimizing risk.

Teams cared about maximizing productivity and their work days not sucking.

As long as teams kept managing to soldier through... rarely saw things change in those shops.

Yep.

mysterious and unsolvable breakage: Helping another startup work through one now. It's a case of reclaiming functionality from a mystery outsourced codebase (without source control) meets inexperienced developers who try their hand at sysadmin plus a 100% rotated bevy of actors (the whole team, PM and all, have jumped ship), no documentation and no technical oversight. Offshore outsourcing adds cultural fun.

unpleasantness: I would expand this to unpleasant or incomprehensible. I have seen projects be de-resourced because of lack of management comprehension when they literally paved the best and most rapid path to profit (later taken successfully by the now-dominant competition).

antiquation: The best example of this I've seen was a hardware product an employer was developing as a joint venture in Taiwan early in my career. Engineers had made the decision to use a sucky chipset from a struggling company to save money, but the supplier went under and the API froze (bugs, missing functionality and all) before our product development could complete. The target feature set was literally impossible to implement on the hardware and nobody wanted ownership. Many millions of USD, wasted.

replacement: It can work out, just infrequently. Generally when it works it's a smaller system with well defined interfaces.

Would these be cases where Robert Martin's 'The Clean Architecture' [1] would help, where the core enterprise logic is separated from third party dependencies, making the latter easy to swap out and replace?

I'd imagine a number of these cases are caused by a heavy reliance on third party technologies that are no longer supported, or very few people still understand.

[1] https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-arc...

It's a great idea but in reality most third-party tools are going to work slightly differently and the abstractions will leak (unless they were developed against an existing interface, in which case you don't need to create the wrappers yourself anyway).

https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...

I think the optimal route is to not bother with (extra) abstractions and interfaces, but try to avoid using things that is unreasonably tied to vendor - unless you save a lot of time.

If the code base is not a pile of dung anyway, the cost of find/replacing and refactoring obsolete or replaced api:s once is so much smaller than the running costs of maintaining an extra layer of leaky abstractions for many years.

It is guaranteed that the abstraction will not work without a lot of changes anyway, and what typically takes the most time is the regression testing.

I've learned over time that its always better to assume that who came before me were smarter than me and knew more than me (I'm rarely proved wrong).

"They did it that way because they where stupid" is a ridiculously common assumption, when the correct answer is often "They did it that way because they knew stuff I don't know".

There's obviously a lot of stuff I don't know then. Like the benefits of copy pasted code, or 300 column lines, or implementing the logic in 20 places when it has existed in the standard library for a decade. If only the ancient sage I inherited this code base from had left notes to guide me on this path of wisdom.

See, taking a reasonable generalization and interpreting it as an absolute statement is a classic case of "assuming they're stupid".

I don't assume people are stupid. I assume they don't know or care how to write code well. Big difference ;)

I've been there a few times, and the worst thing is when every five or ten WTFs there's something that seems as ill thought as the previous couple similar blocks, except this time it actually makes sense, as it implements (awkwardly of course) some important corner case.

Copy-pasted code: Data redundancy

300 column lines: Support for management buying everyone nice, new, giant monitors.

Logic in 20 places: But what if we want subtle differences between each implementation?

On a more serious note, a product that I've worked on was started in about 1998 in C++. We support something like 15 different platforms, and we've got our own implementations of things like vectors because we needed a least common denominator codebase; the standard libraries of a lot of platforms didn't provide what we needed, or provided implementations that were incompatible with other platforms. By the time everything we needed to support was modern enough (in about 2010), the system had a few million lines of code, and replacing things with library functions/classes would've been a nightmare. New development is saner, but the legacy stuff is entrenched.

Libreoffice managed to get rid of their legacy containers and moved to the STL, so maybe you can do it too

https://people.gnome.org/~michael/data/2011-05-12-libre.odp

Development of that particular product moved to China and India last year, so I don't have a part in its development anymore, just build+release, because there are some legal benefits to releasing it from this country.

On the plus side, there are only a few platforms that they still have to support gcc 3.x on, and all the ones that ran on 2.x are out of support (until a customer holds a few million dollars in management's face, as happened a few weeks ago with AIX 5.1).

Well, I think one has to be alert to both possibilities.

This is perhaps a subtle and underappreciated reason that code quality is so important. If you're looking at an obviously well-written piece of code and you see something you don't understand, you can figure it's probably there for a good reason. If the code has visible sloppiness, it's much more difficult to tease apart the good parts from the bad.

Well, the correct answer is often also "They did it that way because it was a reasonable choice then, and they didn't have the benefit of hindsight that I have now."

Its also possible that whatever constraint they were working around no longer exists. Either way, it's a case of not tearing down a fence before you know why it was put up.

G.K. Chesterton, 1929:

>In the matter of reforming things, as distinct from deforming them, there is one plain and simple principle; a principle which will probably be called a paradox. There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.

In my experience Sturgeon's Law usually overrules Chesterton's fence.

"They did it that way because they knew stuff I don't know".

Or "They did what they did because they were the first one's to do it, and it only looks whack in hindsight"

Or "They did what they did because they were living within totally different constraints - like having to support old crap browsers like ie9, or a 'lowest common denominator' of slow end-user PCs etc., or some old chipset on the firmware code they wrote, or some old language paradigm, old libraries etc."

Often it's the opposite.

They did something in an horrible way knowing it was horrible but because they have been asked to deliver a feature as soon as possible and at any cost.

This is a slippery slope. It lets a company move faster till it reaches the point that the software becomes an unmaintainable pile of hacks.

I worked for a company that grew considerably for 10 years and then lost its biggest client and folded quickly. We had spent a few years reworking our platform in a way that might have been successful enough to weather the storm of losing that client, but technical debt really slowed us down.

Technical debt may not have killed the company directly, but we have to wonder how we might have done if we could have spent more of our time on new development.

This is revenue diversification not technical debt. Companies with a single customer funding the business should be actively pursuing a high priority strategy to reduce this risk.

You misunderstand - our software itself was full of technical debt. We spent a lot of time dealing with the consequences of that debt, and I'm wondering if we'd have been able to hit vital targets sooner without it and possibly have survived.

It was both. The combination of two failures put them beyond the point of recovery.

Heh. I have a rule of thumb: any project that starts with the words "this time we're doing it properly" is screwed.