Comment by marginalia_nu | Hacker News Reader

Comment by marginalia_nu | Hacker News Reader

I briefly worked in the (Swedish) public sector, was on a team tasked to replace an old system written partially in Cobol and partially in some IBM-tech nobody understood anymore.

The system was huge and poorly understood, and a lot of people used it to do most of their work. Extremely critical system for the very fabric of society.

There had been several previous attempts at replacing it. They'd sort of fizzled out along the way, so they were stuck with several half-finished iterations of replacements that were in used in parallel with the old system.

This was it though. Hundreds of people. Micro services. Modern JS frameworks. C4 architecture and hexagonal design and cloud native and every goddamn buzzword that's appeared in a programming blog the last 10 years. Just get it all in there! Everything the cool start-up kids were using needed to be apart of this. It wasn't clear exactly how it was going to better, but it sure was gonna be modern.

I didn't see the project through but I could see where it was heading clear as day. It was only a year in and maintenance was already becoming a problem. Shit was falling apart. It was a nightmare to debug. Nobody knew how the heck all the services quite communicated.

I'll eat my shoe if they don't spend the next 25 years fixing leaky pipes with this microservice mess bolted onto the previous iterations as an even more grotesque programming-cronenberg.

I had a similar experience in the Norwegian (hello, neighbour!) public sector, and it was _also_ a rewrite from a partly COBOL-based solution. The difference is that we succeeded, which is - if you're Norwegian - almost unheard of. :)

The whole thing was rebuilt using new tech; Java for the most part, cloud-ish hosting (our "own" bare metal servers and OpenShift via a Norwegian company), and this stack was never a problem. In fact, not much was a problem, because:

The old system was _riddled_ with unit tests that could easily be replicated in any language. All of the bugs and/or minor issues over the last 30 years had been documented and had unit tests covering them. This was what saved the rewrite. Our rewrite could rely a lot on TDD based on those tests, which made it possible for us to trap problems early on.

Without those tests - written by programmers who probably are long dead - the rewrite would never have succeeded, or at least not in the time frame given, and at least not on budget.

Since then, whenever our consulting company is involved in anything that has to do with existing software, we ask the client about how their test coverage is;

* None, but you will give us $1 million per hour? Nah. Doesn't matter. Find someone else.

* Lots, and you'll pay us $1,000 per hour? Tell us more.

So: create (unit) tests for _everything_. Fixed a bug? Create a test for it. Changed some behaviour? Create a test for it. Removed a feature? Create a test for it.

EDIT: Final stack looked like this:

* Self-ish-hosted servers (non-Norwegian cloud providers was not an alternative because of privacy issues).

* Java as main programming language on the backend, but many services was written in C#.

* Frontend didn't have to be fancy, so some simple JS-stuff there; think auto-complete, no fancy DOM-handling or anything like that. Application as a whole was very server-centric.

From what I've heard, they are now trying to get away from Java, and move more and more over to C#. I just hope they keep the unit tests. :)

This is such a good approach. A former colleague of mine built an entire scaffolding of integration tests surrounding a test-free system that needed to have a large fraction rewritten. Mocking clients, collaborators and everything.

The tests were only ever used once for that rewrite (the customer terminated the contract shortly after the rewrite was complete, and the entire codebase was thrown into the rubbish bin) but I still think it was worth it. The tests didn't even run that many times because most of them passed at the first attempt but I suspect the effort of enumerating cases had a strong impact on the successful design of the rewrite.

Spend the first four hours sharpening the axe, and so on.

> Spend the first four hours sharpening the axe,

Great analogy. I'll remember that one.

The first thing I do with any bug report is check for or write a test reproducing it. I’m entirely bewildered when programmers don’t do this. It saves time in so many ways, from explaining PRs to refactors or getting future colleagues up to speed.

Is it ok to ask which company provided hosting?

It's OK to ask, but I hope it's also OK that I won't answer. :) I can't divulge this kind of information on a project that I worked on. NDA etc. etc.

Also, after some DNS-digging just now, it seems like they have changed to a different (Norwegian) provider.

A legacy system with tests? Now that's something you don't hear everyday.

Is it really legacy, if it has tests?

> The old system was _riddled_ with unit tests that could easily be replicated in any language. All of the bugs and/or minor issues over the last 30 years had been documented and had unit tests covering them. This was what saved the rewrite. Our rewrite could rely a lot on TDD based on those tests, which made it possible for us to trap problems early on.

This is the crux of it. The hard part of a rewrite or major refactor is not building out the new stack from scratch using all the latest tech. The hard part is getting the initial system in a place where it can be rewritten. This includes -

- Adding robust testing, both unit and integration/system/functional.

- Working, functional CI/CD pipelines that execute these tests. I have encountered so many legacy projects without CI pipelines, some of which even had tests but were just expected to fail. Not marked in the code as expected to fail, mind you, just expected to fail by developers, who shrugged and only ran tests they thought were relevant to their code change.

- Code cleanup, particularly if you're doing a slow migration by functionality (which you should be). If you're making a "user microservice," you need to structure the code such that all user-related functionality is isolated and written in a way that works as a microservice, before you write a single line of the microservice itself.

- Infrastructure to do parallel implementation[0], including comparison and reporting of differences in results.

The main problem with all of this is that it's boring. Rewrites are almost always, of course, advocated by developers who have to deal with the code on a regular basis. If you tell developers that they can do a rewrite, but first they have to spend 6 months writing an integration testing framework and fix all of the broken unit tests, most won't go for it. They want to jump directly into using Kubernetes on the cloud with Erlang.

The problem compounds when you look at the wider industry, because honestly, it's often a great career move for the developer who starts up a rewrite with Kubernetes and Erlang. They can stick around for 6-18 months, put "Rewrote legacy system using $HYPEDTECH" on their resume, and start shopping around. Doesn't matter if the rewrite ends up being a dumpster fire, which it normally is. That's a much better story for your career, unfortunately, than talking about how you deeply understood the domain and the quirks of the classic ASP code powering your current company's most critical software.

[0] http://sevangelatos.com/john-carmack-on-parallel-implementat...

This is a telling example of the innovation budget idea!

If you're going to replace a creaky old legacy system, never ever do it with the latest technology. Do it with tried and boring technology.

Otherwise soon enough you will have two problems.

I think part of the problem was they needed to pitch the project to the government to get it funded. Probably easier to sell the idea of the Homer Simpson Model S[1] than what they actually needed.

That, and everyone with any sort of deciding power were technology-illiterate bureaucrats who waste no opportunity to try to make us somehow use AI to somehow enhance this system in some fashion, largely so they could announce to the world that their system use AI somehow.

[1] https://imgur.com/97lhWVR

I think the problem is not with the latest technology, the problem is with the inappropriate one.

Have they chose golang, or (God forbid) Rust, they will most likely be fine.

There are choices, which are very dangerous, as they increase complexity dramatically, like micro-services, or "event-driven" architectures, or Kubernetes. Add all three and good luck lifting project off the ground.

One of the things I've had my nose rubbed in several times is just how very effective some of of those "legacy" languages - like Cobol or the various 4GL's - are for writing business systems apps.

It clicked for me when I started thinking of them as DSL's, where the "domain" is "business stuff" ( or equivalent, government data processing etc ) Of course they have limitations - but they're also often part of a full platform and very opinionated as to app and data model, and within those constraints programmers can be crazy productive.

So many of these rewrites fail, and I think a big part of it is people don't fully realise the ramifications of the fact that they're replacing both a platform and a system built on that platform. Sure, there's probably a wizzbang new architecture, which well intentioned smart people will be able to prove improves on that platform in every way - with reams of documentation about the choices made and the way things work... but they're still choices, and sometime they'll get misunderstood, or ignored, or subverted by people who think they know better or who just need a short cut - and it's often big enough project that staff turnover is actually a significant factor too, so opinions may shift. But many of the architecture/data model decisions are just built in to those older languages. The architecture isn't subject to the same drift pressures because they're not choices, and that makes the system less complicated overall, and especially over time.

Of course limitations of the various language implementations, as well as the very human distaste of working within a constrained system is probably a big part of the reason that these legacy languages effectively died.

But I think we lost something valuable, without many of us ever realising.

Reply on news.ycombinator.com