Aging Canadian government computer systems at risk of 'critical failure'
cbc.ca
cbc.ca
I initially put all of the blame on the expensive consultants, that were doing most of the work, for being technically awful (writing software like Fowler's Analysis Pattern book was a holy grail).
While I still think they bear the brunt of the blame, in retrospect everything was awful. The project was both a business and technical transformation at once. There were too many managers, too many business experts and overall poor leadership. Rather than breaking it down into manageable chunks, it just kept growing and growing. There was no concern or accountability for waste.
Sometimes replacing a legacy system with a new system is difficult. But most of the time, it can be done as a slow and steady transition...so long as egos and ambitions stay out of it.
What I've noticed on the ones that work is that they have a core group of technical managers and architects that hold in their heads the whole system architecture and organization. These are people that for more than 10 years decide what will be done and how it will be done. This seems to happen on the more financially crucial and essential areas of the government (e.g: revenue from resources). It is in the more accessory/non-essential departments where there is rotation of technical staff that the chaos thrives.
Profit centers are usually organized and efficient. Cost centers aren't.
If a new-ish web app for a bank or government service has a “maintenance period” where it becomes inaccessible between 5PM and 9AM each day—without the actual web layer going out, just seemingly changing its access ACLs on the data layer—that’s because they’ve got the web app fronting a mainframe that was designed for explicit OLTP ingest / OLAP batch ETL phase separation. The new record event stream gets collated at night to build queriable/indexed tables. Often on hardware so slow that it needs the entire maintenance period to finish building those tables+indices. (This is why better performance in these systems is hard: if you want higher OLTP ingest throughput on the same hardware, you can have it, but then they’ll need longer maintenance windows, so long that they’ll now break their SLAs by being unavailable during business hours!)
I've quasi-blogged at https://old.reddit.com/r/dredmorbius When I do re-start, that will at least get updates.
There's a new blog, I promise, eventually coming to GitLab, likely:
https://gitlab.com/dredmorbius/lair-of-the-id
Or https://gitlab.com/dredmorbius/synopticon
And yes, technology as predicated on energy flux is a principle I've largely accepted. Vaclav Smil's work is singular in this, though I find Daniel Yergin's to be so as well, though that seems largely unintentional / accidental.
It's oil, not money, that changes everything.
(Or at least changed it.)
I feel like there’s something to be said for constructing an architecture entirely out of “formalized” architectural patterns, when you know that 1. you’re not going to be around to explain the architecture to whoever has to maintain the project 20 years down the line, and 2. Anyone who does come onto the project to maintain it is going to have very little time to get up to speed, since they’re very likely on a short contract to implement a single fix (“Y2K compliance” being a good example of when this happened.) At least, if all the architectural arrangements are copied out of a book, you can just put a note in a README at the project root to read said book. Might skip a lot of “software forensic anthropology.”
I mean, that, or heavily document every piece of the architecture, to the point of writing a book on your architectural choices (e.g. the reams of documentation on Erlang’s OTP-framework architecture, originally produced as a side-effect of Ericsson engineers documenting their architectural choices while writing code for their network switches.) But, well, did the customer pay for that? No? Too bad.
(I find this is one of the major differences between the way a regular IT consultancy handles these kinds of contracts, and the way a bigcorp like IBM handles a contract. The bigcorps factor things like “reams of documentation for later maintenance (not necessarily by us)” into their negotiated price, because they’ve been on both sides of that, and also have had customers who have been burned by a lack of it. But this makes their bids higher, so they’re often underbid by novice consultancies who don’t have share this mindset...)
Or the bid has no clause about documentation at all!
That Phoenix pay system was a total cluster that lasted for years. I remember reading headlines and thinking “they haven’t fix it yet?”.
Made the ACA website in the US look like a resounding success. People were literally not getting paid for months (years?) and the gov’ts was like “working on it!”.
One of her coworkers hasn't been paid in almost 6 months.
How many people can go months without a paycheck.
This is Canadian Government Bureaucracy in a nutshell. Decades of (often minor) scandals have caused successive Governments to restructure in such a way that it is nigh-impossible to assign responsibility; and have done so by making it extraordinary difficult to form clear direction and make decisions.
What we need is a massive downsizing of the public service at the managerial level, and begin placing the burden of responsibility upon departments. They need both the power to make decisions and the associated risk of failure; whereas now the decision making power is nigh-ephemeral and the risk is distributed broadly and thinly.
Managers, subject matter experts and consultants are paid by the hour unfortunately, so the incentive is not to deliver the project, but to instead make it take as long as possible.
Of course building up a good internal team inside a government department also seems to be frought with peril, so maybe it's the only option.
When the top 1/3 of the org chart is gonna get fired by the next administration regardless there's no reason (for the current administration) to hold them accountable for not doing good work (not to mention that most people aren't going to be harsh to their own political appointees) and that sort of culture rolls down hill really well.
The problem is these big projects, like modernizing core social service systems, are given big budgets and big expectations. Then they start a big team to work on it, with short deadlines and not enough direction. At some point it is discovered that the legacy systems business logic is mostly undocumented/baked into a mainframe/functioning through sheer luck/nothing but PL/SQL and DB triggers. Then the several contacting companies are brought in to try and sort out the mess, solutions are purchased from billion dollar private companies (all of which have screwed over the Government of Canada one way or another), and project management is shuffled around as fast as HR can fill out the forms. After three years the money runs out, the project dies, and then we try again in two years.
The solution is never a slow and steady modernization. It's never small, focused teams working on improving or rebuilding individual components. It's never upgrading infrastructure, or moving to modern best practices.
It's always all or nothing. Solve the problems, without fixing them.
I recall one fairly large project where the Director of an affected agency would nix any changes that affected their agency. Period. Status quo was sacrosanct.
Part of the equation is that the staff have a very strong union and very few in Gov't are willing to make any change that will impact unionized staff (i.e. becoming more efficient is frowned upon).
Combine all of this with the (fairly opaque) hidden agenda of consulting agencies to keep the project going for as-long-as-possible and it's just a recipe for disaster.
Always was and always will be.
/rant
Only with a clear idea of what things should look like can the new structure be built, tested, (and while testing) training written and validated, and then a rollout planned (which is it's own whole other project).
Because my personnal intuition would be that trying to do both a revamp of the process, as well as performing the technical migration is the actual recipe for disaster.
I would do the migration while remaining a close as possible to the original system (removing the obvious unused functions), and only then start transforming the business process.
The desire for very little change (which is difficult and does need someone to push the politics of that change through) would leave us in the belt and pulley driven workshop layout making horse and carriage gear.
( Ref: https://en.wikipedia.org/wiki/Electrification#Benefits_of_el... )
Also considering forward thinking, this is baseless speculation but, my gut feeling is that science fiction written today is more likely to be 'close enough' to how every day computers might work in the future that such systems would seem like a plausible alternate reality. Contrast that with what science fiction written even 50 years ago thinks about anything involving computers or automation.
That's why it seems likely that the overall workflow should be examined again including a look at what is actually needed and what tools we currently or might have to accomplish those tasks. The existing systems, interfaces, and forms are __some__ of the tools to consider, but if there are actually good reasons for evolving or replacing them those changes should be documented and made.
What we don't do is look at the old code, document how it works, and then reproduce that. Prior to this legacy system being built the entire company worked with paper processes and documentation, so it was a paradigm shift for how the business worked. That system is slow to update, so how they work is heavily influenced by the business' thinking 40 years ago. Our replatforming project is seen as essential for the business' continual survival, so we're allowed to question processes, simplify where we can, and work as equals with the business in defining new processes. There are definitely hold-outs and resistance from some quarters, but once you launch some successes people start converting and accepting the process.
it's also probably very different whether the system has been continuously adjusted and fixed for 40 years, or if it's just stayed there.
For example, what was once a file based batch process meant that other processes had to wait and split their processes accordingly. Replace that with an event based system and everything can run contiously and a lot of the restrictions disappear.
Modernization to me (specifically in the context of the GC) means refactoring/rewriting systems to follow modern industry standards and practices. Things like proper version control, automated testing, automated/semi-automated deployments, and monitoring and logging.
I think there are major benefits in adopting these practices for both the Canadian public and the developers building/maintaining the solutions. Unfortunately, getting a budget to move the source code from a shared network drive to Git is next to impossible. And God forbid you want to spend time adding any sort of testing.
At some point, someone gives IBM or various consultants millions of dollars to implement some IBM software--Powerbuilder, Java, Domino, etc.--which they do, following RUP, which means a pile of UML docs on top of it. It actually works, the gov't says "great, we can stop funding ongoing dev for a couple decades."
Two decades later, CGI has taken over the IBM project for half the maintenance fees because they know how to do RUP, and thus meet the piles of NFRs IBM left behind (such as "all metadata changes must be auditable", meaning all UML diagrams must have a documented review/approval chain separate from source control").
I sat in a meeting where a stakeholder showed us the report generation page, and said "if we click on this link, it crashes the server, and we have to open a ticket to restart it".
Software does rot, and gov't procurement processes and requirements are those left behind by IBM to prevent smaller, "agile" agencies from doing meaningful work. To my mind, there are two flaws here: one is letting IBM in, in the first place, to set the ongoing standard; two is refusing to acknowledge that every organization needs an organic software dev capability that can use outsourcing as a resource multiplier, not as an IT replacement.
Two anecdotes:
1) I know someone who deals with legal documents for the Canadian federal government and she has multiple word processors (of various versions) for these documents. Since minor changes can affect the meaning of these agreements, they are only edited or amended in the software in which they were originally created. She is understandably very excited for when the ones created in old versions of Word Perfect expire and a new agreement is created in newer software.
2) Only in 2019 did the US Nuclear Command transition away from their own 50 year old technology: floppy discs from the '70s. [1]
It always fascinates me to see what sticks around, for what reasons, and how it affects people's work. I've heard stories of developers creating fancy UIs to cover up ancient Fortran software, so that it's less painful to work with, but they don't need to replace the underlying software.
[1] https://www.nytimes.com/2019/10/24/us/nuclear-weapons-floppy...
Once installed in the Ontario government, there was kind of a divide between people who wanted to move on up to the Federal government (seemingly they were not concerned about skills not being transferrable, hence my guess on COBOL and old mainframes being in use), and the other half who wanted to just put in their time in one place and collect their paycheque and pension.
Also, can confirm on the expensive contractors. IBM in there pushing Rational Rose and hardcore waterfall did nobody any favours at all.
Exactly. My first job out of university in 1996 was with the Canadian government. One of my first tasks was to update a COBOL program that had originally been written in 1972. I left that job in 1997, but I doubt the program I updated has changed much in the 24 years since, so it's at least 48 years old at this point.
An API is an API. As long as the interface is well designed, my POST request can release a carrier pigeon on the backend for all I care.
From my reading of the article, it appears they are still using 1970s era IBM Series/1 minicomputers. It sounds like all they've done, is replaced the physical floppy drive with a floppy emulator. This is a device which attaches to the legacy IO bus, and appears as a floppy disk drive to the minicomputer, but the actual data is stored in flash memory instead of magnetic media.
If you go to a candy-store at the beach and see a 150 year old saltwater taffy machine, you say "ooh, cool". The thing that makes a missile go up is essentially the same thing, it needs perform it's function to spec, period.
I know someone who works for an industrial gear company and routinely reconditions things like draw bridge mechanisms or a rubber rolling mill (do they call it a mill?) and most customers just want things to keep on working like they've always been working since they've built their process around them and machine throughput is rarely the bottle neck. The improvements customers go for are modern bearings (quiet and cheaply serviceable) and if they have to have gears made they generally opt for a more modern tooth profile (potentially stronger, quieter and more efficient).
How much difference can switching from word 2007 to word 2016 be? Even switching between microsoft office and libre office, the only thing that would get mangled is the formatting, not the text itself.
I would guess it's mostly paranoia--you'd have a hard time convincing a judge that the one orphan item that spilled over onto page Y+1 wasn't meant to be included--but I can sorta see not wanting to risk it.
I mentioned that they should replace that thing sooner rather than later, but their objection made sense for their use case: The replacement would be many thousands of dollars and this old setup still worked. The computer had one job, and it did it. Who knows, maybe it's still there, still mashing away on its floppies every day. It wouldn't surprise me. If it is, it's being held together by nicotine deposits and luck.
This is where Western Governments are actually corrupt, but it doesn't show up in Corruption Transparency Index.
I'm not ideologically a 'small government' person, but I have absolutely no faith in our government's ability to do anything reasonable in IT.
Sometimes I think we need 'government in a box' IT solutions. Sadly, even if we did, they'd still labour over them in some way and make it expensive: the whole point is for vast cadres of the civil services, and consultant/lawyers etc. to suck money out of the system.
The big problem is that these projects are managed by people who aren't technical, who have never built anything concrete and whose gulf of cultural experience between manager and worker is enormous.
In the rest of the world this is being fixed by programmers drifting out of the enterprise and into dedicated software firms. The non-technical people in "the business" get to issue RFPs, watch slide shows and be in meetings: where they like to be. They cut a cheque and get a system they know works, they know what it does, it has a predictable price, it's a web app so it bypasses their terrible IT department. The technical people get to work for each other and bosses who are themselves former programmers, so they don't get asked every five minutes to draw Gantt charts with to-the-day ticket time estimates. Everyone is happy.
That hasn't really happened with government, probably because there are so few of them. Government IT in a box is a great idea though.
Most companies that screw things up that bad, will fail. If they don't, it's still their right to waste money on dumb projects - it's their money.
Government failures (at least in Canada) generally exceed those in the private sector for that reason, exposing the dire systematic problem if 'no competition, no oversight, lack of competency' on a scale rarely seen elsewhere.
Not only is there 'no incentive' to fix problems, often there's also a negative incentive.
It's 2020. The technology to put my medical history online has been available for 20 years. Ontario, Quebec etc. have still completely failed to do this. I still have no easy way of finding out which clinics are available for me, and when I do go to a new one, they have to open an entirely new file, totally unaware of my historical medical issues. To make matters worse, it's literally illegal for me to pay anyone to provide me with medical services. It's kafkaesque.
A very basic medical history system, that merely documented doctors notes etc. could be done 'on the cheap' (relatively speaking) - but it's far from happening.
Even an intelligent regulatory mandate could solve the medical records issue, i.e. providers must participate in XYZ system, with ABC components, designated by the government. But we can't even have that.
It's really bad and I don't see any path to getting better until government develops a whole new attitude towards IT.
Granted, it is all happening slowly. I worked in ehealth in Ontario around 2000 and they were providing huge incentives for organizations to go digital, but most didn't want to due to habit, and because each institution is run as an empire.
And ultimately, the results from each system, even if it's the same test, are often not really comparable, so the dreams of results that can be compared by the consumer, and precise reasoning systems for actual AI are another generation after consolidation.
1. https://www.quebec.ca/en/health/your-health-information/queb...
The only difference is the governance structure. In .gov, you tend to have professional / civil service people at the senior director level who know their business inside and out, with a political layer of management who drive change and vary in competence.
Medical records are a great example of how .gov/.com doesn't matter. When stuff gets complex, IT sucks.
I don't agree at all.
Big corporations can fail to do many things where it's not really important, so it might seem like 'failure' but mostly it's a function of market conditions. Other big failures (say Boeing 777) are understandable due to complexity.
Very few groups on earth can build such airplanes.
Anyone can build a gun registry.
I loathe how long it takes my bank teller to speak to me, but my banking services are in the end, amazingly cheap for what they provide.
Governments do a reasonable job at things like contract allocation for road maintenance, some kinds of construction, but they generally do a bad job operationalizing anything.
This is not at all true in Canada. I'm really confused by this statement.
If you start to provide services normally covered by the government, you will be shut down, or you'll have to take your case to the Supreme Court where this law is still being tested.
There are places that provide parallel services, they operate in a grey area. For example, the Supreme Court of Quebec ruled that private services can be provided for treatments wherein the government does not provide 'timely service' i.e. 'wait times are too long'. But exactly the parameters of those 'wait times' nobody knows, and the only way to find out is to go to the Supreme Court. So not a good business plan.
The laws vary from province to province however.
[1] https://www.canada.ca/en/health-canada/services/health-care-...
I have heard of systems with five layers of emulation, yet still quite a lot faster than the original machine.
I gather Bloomberg has begun to do this, emulating SPARC user-space on Intel so they can stop paying Oracle to support long-since EOL'd 32-bit equipment and OS. (Or maybe some sort of hybrid.)
I can understand why no expensive consultant would suggest it; it is very cheap, and quick to set up. No fat contracts there.
Neat to see they've got some of their work open sourced: https://github.com/cds-snc
Finally went there for a sofware event and people in Canada are sweet, calm, happy and non-stressed.
I've heard many stories where the contractor knew the job would have to be done twice the moment they read the procurement documents. But they couldn't voice their concerns. And if they suggested doing what they knew was the right thing that would have made them ineligible for the contract as it wasn't what was required. Future-proofing the bid or trying to deliver something closer to what they ended-up shipping was also not possible because this would have made them more expensive than the bidders following exactly the request. In the end they ended-up rewriting most of the code at their usual billing rate on top of the original fixed cost contract.
In the case of Phoenix, I've read a lot of media articles outlining how bad of a job IBM did but despite all this it seems the contract itself was never challenged in court. What I heard from internal sources is that IBM did ship correctly all what was asked for in the contract, it's just that the government workers drafting the requirements didn't understand their own payroll needs enough to properly articulate them.
Of course CBC, the state-funded media where everyone is on the government's payroll, won't outright blame their bosses. But they would get sued and lose pretty bad in court if they claimed the contractors didn't deliver what was in the contract. So you get articles with a weird spin where they try to blame the contractors without going too far and paint the government as a victim.
Years ago, I had a past employer that would immediately toss resumes from applicants that had recent employment with the government. Yikes.