Tell HN: Canada Posts systems to be down for 6 hours of “unplanned maintenance”
status.canadapost.ca
status.canadapost.ca
They know 80's mainframe could do updates without any downtime right? Did someone mess up the Kubernetes config and shut it all down at the same time?
The Canadian government is an example of how not to do software. Their Phoenix pay system is frequently used as a cautionnary tale for bad design and implementation [0].
Ouch.
And for those who think software engineering other than medical devices, aviation, or nuclear power plants isn't life-or-death:
> Pascale Boulay, a Quebec coroner, determined that the 2017 death by suicide of a 52-year old woman from Val-des-Monts, Quebec, was preventable. The coroner assigned blame on the "flawed Phoenix pay system" that had "led her to emotional and financial ruin."
Red flags independent of software development finesse or skill:
- Centralized payroll operations to effectively the middle of no where (effectively ensuring that no existing payroll specialists would staff up the new center). Locate operations in the middle of no where to compensate for another government program and center being shut-down.
- Sole source contract. To IBM no less. To customize and configure PeopleSoft. There's literally nothing that sends good vibes in that setup.
- The very concept of running a big bang switch over of like 75% of the federal government to a new payroll system.
- Launched without end to end, full system test
- Launched KNOWING that critical functionality (like retroactive pay) was not implemented, and that critical requirements were failing tests.
- Deciding to roll out despite their new centralized payroll center was already less efficient at handling payroll on legacy systems and that their central assumptions were already being challenged
- Deciding to roll out despite receiving reports from client departments that the initial data transfer into the new system was already significantly incorrect
- Quote: "In our opinion, they had received more than enough information and warning that Phoenix was not ready to be implemented, and therefore, they should not have proceeded as planned. Phoenix executives prioritized meeting schedule and cost over other critical elements, such as functionality and security, resulting in an incomprehensible failure of project management and oversight"
You can read the audit here: https://www.oag-bvg.gc.ca/internet/english/parl_oag_201805_0...
Software is only tangentially related in that the "magic" of software lets people believe that such projects are even possible.
I don't think a single real engineer was involved in this, sadly. Probably offshored contractors and clueless non-technical government employees.
It's ironic because Quebec punch way above it's weight for engineering.
Not only did they push it through despite issues even after initial limited rollout, they made a ton of terrible decisions like to do training in house instead of from IBM as cost savings, made the new Public Service Pay Centre elsewhere for political reasons with new staff and they laid off most of the people that worked on the old system and decommissioned it so there was no going back.
My understanding is that it wasn't IBM's fault at all. They delivered exactly what was aggreed upon. What the government asked made no sense.
That's why they couldn't and didn't sue, and if you read about it in the media, there's no direct blame on IBM (a lot of it is implied; being to direct could be construed as defamation).
If what the government asked made no sense, why did IBM agree to do it? Rhetorical question. Money talks.
My understanding is that the government gave IBM a complete spec of what they believed were their pay grades. But that wasn't what was agreed upon with the employees and unions (how do you calculate overtime on a coast guard boat?). So it all made sense to IBM, they shipped it and fufilled every requirements. They had no way of knowing these weren't the "correct" pay rules.
It's the magic of government contracting: you write the spec and select the lowest bidder.
Forget a 0 to the total weight the bridge must carry? You'll get very cheap proposals. Ohh and there's typically rules against a bidder contacting the government to get more information; that could be considered as interfering with the bidding process. It's also why older USPS vehicules had poor snow performance.
In this case I think all her income was from a single source (the federal government). So it doesn't sound political to me as they were able to root cause the single source of financial ruin.
It's like naming the drunk driver who slammed into a pedestrian while being completely alone on the road. You can always say "an anonymous driver" on the report ...
And yes, Phoenix was awful, but separate point.
The only two options to contact Canada Post were phone or … Twitter — which I don't have an account for — no email or web form. And the phone CSR had to get a manager who still couldn't figure out how to deal with the issue since I wasn't a customer myself and just wanted my email removed from the account.
I forgot to say that another option was to contact them by mail, so at least they're dogfooding ;-)
Here in Hong Kong English supercedes if presents and next door in Shenzhen it's the opposite.
Where this approach falls flat is web domains. It requires you to actually type out both the language you know; along with a language you (likely) don't. Quick Google only 1.2% know both.
And as pointed out above, then figure out if those variations are separated by a dot, underscore or hyphen. Notwithstanding the domains get ridiculously long.
It's good we try to please everyone, but sometimes the politeness goes too far. Registering the domains in both French and English in this case should have been more than sufficient.
Any logical personal will understand showing content in multiple languages (as text output); but when it comes to user text input (domains, web search etc), it's absurd to believe the same rules should apply for the user.
That's the price of free trade and open borders. Most countries lose their best and brightest to the US and other high-pay countries, and the world is poorer for it.
Its not SF levels of pay, but its definitely not peanuts.
Why not simply compete?
Yet it's a single payer healthcare system no? Meaning medical staff ultimately is on government payroll. Pretty sure they make more than $200K.
> he already wants to defund the Canadian Broadcasting Corporation, also a crown corp.
He's not wrong. Why does entertainment need public funding? Marvel and Disney made billions last year by simply making content that people want to watch. It's a hugely profitable business.
> Our crown corps also happened to be big fans of using students: https://www.lcbonext.com/
The link you shared is even more bizarre. Almost at the top of the page they brag about the companies where their former interns ended up. Most of them are American companies in SV. What does that tell us about the people working there full time? "Not good enough to come to America and innovate? Join full-time with the rest of the B-Players!"
You know that CBC isn't all about entertainment, but also hires a lot of journalists, right? And I don't see Disney or Marvel making content with the Canadian culture and heritage in mind.
That... weird. Having government approved journalists on government payroll sure sounds like something out of China, Russia or Cuba.
I mean let's face it, will they really critisize and investigate the hands that signs their check?
It also has the second order effect of making it harder for independant medias to compete, since they can't just get free government money.
> And I don't see Disney or Marvel making content with the Canadian culture and heritage in mind.
Is there a market for it? Right now this content is dumped by a single player that's subsidized. No reason to enter this space.
The government doesn't actively handle the hiring, CBC has journalistic independence from the government. They in fact handle all of their internal systems (email, data, phone) independently from the government.
> I mean let's face it, will they really critisize and investigate the hands that signs their check?
They do all the time in fact, and they report on themselves even if it shines a negative light.
Example: https://www.cbc.ca/news/canada/khan-cbc-canadaland-tweet-che...
> Is there a market for it? Right now this content is dumped by a single player that's subsidized. No reason to enter this space.
When the majority of the private content available is from the US, it's also hard to compete with the torrents of culture they push on us. At least CBC is actively pushing content that is culturally significant.
But, at the end of the year, who signs the check?
> it's also hard to compete with the torrents of culture they push on us
Not sure "pushing" is the correct term. I remember when Hulu and Netflix started to disrupt the market a lot of Canadians were trying to trick those companies into serving them, with fake addresses and VPNs.
Didn't radio stations used to play "Government mandated Canadian Content" at night because listeners were just tuning out if they did it during peak hours? [0]
[0] https://en.wikipedia.org/wiki/Canadian_content#Regulations
My only nit pick on the CBC is that I believe the very REASON Canada has so much fantastic media is because the government funds independent artists, could someone like Matt Dusk have had a career in any other country? He certainly doesn't say so. Leonard Cohen, Drake, Shania Twain, all funded by the government, because of the Canadian Content mandate, the mandate dutifully fulfilled by our national broadcaster. Additionally, I very much enjoy shows like DNTO, Quirks & Quarks and especially: not having them chocked full of advertising.
Most newspapers are all copy/paste nationwide now, and the majority of TV production is handled by the people who own the medium into the house (cable/fibre/satellite), making "Canadian, eh!" versions of American culture and passing it off as CanCon.
The CBC are out there at least trying to keep it real. Most of them, at least.
It's interesting to consider Justin Bieber's houses right here in California were purchased in part with Canadian government subsidies [0].
[0] https://www.velvetropes.com/backstage/justin-bieber-house
They aren't government employees though. They all bill the same insurance, but they still have to compete on service, efficiency, availability, and friendliness. So inevitably some healthcare providers will do better than others.
> Why does entertainment need public funding?
Because it's more than just "entertainment".
> Marvel and Disney made billions last year by simply making content that people want to watch
And now you have powerful forces who want to perma-extend copyright terms and lock up popular culture behind paywalls until everyone who cares about it is dead. And even longer than that.
> What does that tell us about the people working there full time?
In professional sports, it's common for some teams to advertise themselves as feeder teams for the big names. The pitch is "Join us and see where you could end up one day."
That sounds like using contractors to skirt employment laws for public servants.
> Because it's more than just "entertainment".
sitcoms that end up on netflix?
> In professional sports, it's common for some teams to advertise themselves as feeder teams for the big names. The pitch is "Join us and see where you could end up one day."
Again, what does that tells us about those who join those teams but can't transfer out? They weren't good enough for the real teams.
Could you picture Google bragging about their interns going to Apple or Tesla?
Not sure which employment laws are being skirted here but whatevs.
Defence companies typically also have the same buyers as healthcare providers - governments. Unlike healthcare providers, they're even more limited in who they can sell to. So whatever you're saying applies to them too.
> sitcoms that end up on netflix?
First off, cultural exports are a good thing to have. But also, the CBC does other valuable work. Other people in this thread have also explained this to you but it seems like your mind is made up, so I won't bother.
> They weren't good enough for the real teams.
Yeah maybe, so what? No shame in that; everyone's gotta work somewhere. You're also ruling out people who are there for personal reasons, or because they genuinely like the place, or they don't have any "higher" aims.
> Could you picture Google bragging about their interns going to Apple or Tesla?
Last time I checked, Apple or Tesla (or Google) weren't planning to sell liquor in Ontario. They're not competitors.
No shame indeed. It's just weird that they brag about their best interns leaving for better companies. Usually, the point of having internships is to attract and retain talent.
> Last time I checked, Apple or Tesla (or Google) weren't planning to sell liquor in Ontario. They're not competitors.
They are when it comes to tech hiring. By their own admission, they hire from the same pool.
You can get "level 6" salaries as a new grad at a canadian bank.
To suggest that Samoa, Bangladesh, Peru, or 150 other countries should "simply compete" shows a complete lack of knowledge about the state of the world, about economics, and about human behaviour.
You've never been around athletes haven't you? There are a lot of very talented people who are working very hard to shatter his records!
> To suggest that Samoa, Bangladesh, Peru, or 150 other countries should "simply compete" shows a complete lack of knowledge about the state of the world, about economics, and about human behaviour.
Sounds like they should understand why contributors are leaving and take measures to simply make them stay. Every country will have different challenge, but it's not hopeless. Keep in mind Japan and South Korea were extremely poor a few decades ago; look at them now!
At the end of the day, I'm on the other side of the transaction (getting an influx of foreing educated professionals willing to come here and innovate) so it doesn't matter too much to me really.
Yeah, and a handful can compete with him. 99.99999% of humanity wouldn't stand a chance.
> Sounds like they should understand why contributors are leaving and take measures to simply make them stay. Every country will have different challenge, but it's not hopeless. Keep in mind Japan and South Korea were extremely poor a few decades ago; look at them now!
You named two countries. There are 200 others.