Hacker News Reader

Comment by chx | Hacker News Reader

Botching a prod update -- that happens to the best of us. And sorry to say I doubt Rogers would attract the best of us but I digress. But only kind of digress.

When the human waste hits the circular cooler device, that's when we find out how well an organization is built and managed.

Who is surprised here it took them this long to recover? The necessary no-blame culture, temporary removal of decision barriers, all-hands-on-deck to get back to normal, all the things the lucky of us take for granted are guaranteed to be missing here.

View on HN

> Who is surprised here it took them this long to recover?

They have no independent management infrastructure. Their management network rides logically on top of their main physical network. So when the latter went down you probably had people going to the physical locations of various DCs. But then you have to coördinate between them, and if your employees use your own cell phone service for communication, and that's down…

So the first couple of hours was figuring out what went wrong, then it was having people scramble to locations to plug in serial consoles, then it was buying a bunch of SIM card from other companies, and finally folks managed to wrap their heads around the big picture and come up with a way to bootstrap the network.

An independent management infra would also help to assess as well as recover. There might be a need to require either such an independent network, or one that fully does not reuse your own internal network as a backup link for the management interface.

> buying a bunch of SIM card from other companies

Sounds like some employees already had those for such an event, which is impressive. From the article:

> some employees started swapping out their SIM cards for Bell or Telus SIM cards that they had received back in 2015 as part of an emergency contingency plan established between the wireless carriers.

> And sorry to say I doubt Rogers would attract the best of us but I digress

"The typical Rogers Communications Systems Administrator salary is $67,329" (https://www.glassdoor.ca/Salary/Rogers-Communications-System...)

"The average system administrator salary in Canada is $71,981 per year or $36.91 per hour. Entry-level positions start at $61,425 per year, while most experienced workers make up to $94,400 per year." (https://ca.talent.com/salary?job=system+administrator)

Maybe related are the exceptions to labour laws while you make your 67K a year

https://www.ontario.ca/document/industries-and-jobs-exemptio...

But I'm sure Rogers would never abuse any of those.

That's not a lot of money for software folks. Especially folks with access to the American labour market.

Note that $1 CAD is also $0.80 USD, which is often omitted but meaningful when comparing pay just over the border.

And taxes, goods, and services are a little more expensive in Canada.

and traveling overseas

Is it? Flights from Canada to Europe or Asia can be cost-effective. It's domestic flights and CAN<->USA flights that can cost you.

They can be cost effective only if you can travel in quiet parts of the year. I was also mainly referring to buying things in Europe/US/UK with a salary paid in a weaker currency.

Median house price in Vancouver - $1,341,643

You could move to Toronto where the average house price is said to be $1,167,000.

Ottawa is cheap at around $965,000.

I doubt any of the IT infrastructure in Canada is build by the “best of us”. And it shows, banking, healthcare, telecom, they are all stuck somewhere in the 90s.

The best of us can't be bothered with those insignificant things! They're too busy building algorithms to squeeze every penny out of advertisement dollars on social media.

And can you blame them? That's where the money is.

And even if you are into charity, it's usually more efficient for you as an individual to make more money and then donate to whatever GiweWell lists at the top of their list of most bang-for-your-back charities. Instead of working in a soup kitchen, or a Canadian bank.

oh I am absolutely not blaming anyone! The free market is an efficient machine in this case. It simply means that there is room for innovation in those industries like healthcare and banking. We have already started to see this innovation with companies like One Medical and Square.

If only telecommunications was a free market.

In general, I found it helpful to not look at 'free markets' as a binary Yes/No thing, but as a continuum with lots of different aspects. Free markets are best, but markets that are only slightly freer are (typically) already slightly better. Incremental improvements help.

For example, mobile phone connectivity is still very regulated, but in many countries competition seems to be much fiercer there than for home broadband.

> I found it helpful to not look at 'free markets' as a binary Yes/No thing, but as a continuum with lots of different aspects.

I am curious. What are some of those aspects?

I'm not sure I have a general, abstract answer. But here's a few case studies:

https://en.wikipedia.org/wiki/Impact_of_the_privatisation_of...

About the last: the (partial) privatisation of British Rail was a mess. Far from a textbook case. But still ridership numbers that had been in seeming terminal decline picked up in absolute terms, and rail's relative share of the overall transportation market increases total.

https://www.econlib.org/library/Enc1/AirlineDeregulation.htm...

The impact of deregulation of (American) airlines has been tremendous. Flying is now cheaper than ever. But to address my point: flying is still very strictly regulated, just less so than before (especially in the areas of pricing and competition).

https://www.econlib.org/library/Enc/TruckingDeregulation.htm...

https://www.econlib.org/library/Enc/SurfaceFreightTransporta...

Despite common complaints on the Internet, the US is actually still really, really good at running railroads. It's just that their area of competence is freight rail. Passenger rail in the US is anemic.

Guess which part is in government hands, and which one is comparatively free market and less regulated? However, neither area is completely free market or completely state controlled.

Another example: a few years ago Germany legalized long distance busses. The old ban was a hangover from the Nazi era.

https://www.dw.com/en/regulations-eased-on-long-distance-bus... is an overview article from just before the legalization of busses came into effect. I leave it as an exercise for the reader to find some sources from afterwards. Google Translate might be helpful, if you don't read German.

And, of course, legalization and deregulation here just mean different (and a bit less) regulation. There's still plenty of rules. We are talking about Germany, after all.

Isn't it funny, though, how many people say they can't help working soulless jobs because the capitalist machine controls their lives, but they also say they refuse to feel the least bit invested in their jobs because because they don't let their lives be controlled by the capitalist machine?

Working purely for money and being emotionally detached from your work is one of the bad consequence of capitalism -- in fact it's the most hellish consequence of capitalism that most people working software development jobs are likely to personally experience. I don't understand why people working at adtech companies talk about being emotionally checked out all day as a healthy way to adapt to the system.

I grew up in East Germany. Socialist workers' paradise!

I can tell you that jobs sucked compared to comparatively capitalist West Germany.

(Basically, capitalism might or might not be bad; but incremental improvements to it have a much better track record that trying to switch to a completely different system.)

I agree! I work for a company that makes a product that I think is a positive force in the world rather than a destructive one. I work for less money than I could at Google or Facebook or Amazon and allow myself to get emotionally invested in the work, and I think having the opportunity to make that choice has made my life much better. I know this is a luxury that not everybody gets under capitalism, but I'm always confused that so many people who could have it willingly pass it up to make more money doing a job that they can only justify by pretending they don't have a choice.

Do keep in mind that people also love to complain. That's just human nature.

I used to work for Google for a while. The gig was pretty cushy, but I could have still rattled off lots of complaints, if you asked me.

Almost any programmer job is almost infinitely better than what most normal people have to put up with. And better paid.

I like OneMedical (I'm a customer) but I don't know how much "innovation" it's done in healthcare. That's not to say it's their fault, healthcare is a startup graveyard, but you can't really go to OneMedical for anything beyond the most trivial, and you can book some stuff / send messages using an app.

Kaiser arguably did more innovation than OneMedical, as far as I can tell.

I am a dev with 4 years experience. I haven’t been able to get a bank, telecom, or especially health care org to offer me more than 80K Canadian a year. 64K USD.

These are sleepy places where you go to work 15 hours a week and on demanding days call in sick.

It’s actually worse than that. I would hardly count myself among the best, but I’m certainly experienced and would absolutely love to try and fix the absolutely appalling state of health care records for example.

But besides there being no money in it, between regulatory hurdles, fragmented ecosystems and lack of incentives for individual doctors or hospitals to be good at sharing or maintaining these types of records, I could give the software away for free and still fail to gain traction.

In all of these spaces I don’t see how it progresses without government regulation and/or incentives, because the incumbents have literally no reason to improve.

Yeah, because they probably still pay like it's the 90s.

I think there are more fundamental issues - especially in healthcare infrastructure.

For the Phoenix government payroll system, they paid IBM like it was the 2090s and it’s still broken.

having to deal with legacy systems and interoparabality is a significantly bigger issue than developer pay as evidenced by the very well working digital infrastructure in much poorer countries that just leapfrogged the old stuff.

But in poorer countries, those companies probably pay relatively well. Sure they make less than Canadian developers at big banks, but they relatively make more so they would be better in many cases.

they pay relatively well in Canada too, and for arguments sake, the US which has probably the best paid devs across the board has a financial infrastructure from hell when it comes to payments or money transfers.

It's just tech debt. If you've ever been to China, it's crazy how well WeChat works. There's just nothing like it in the EU or the US because we have 20 different decades old things for everything that people can't or won't replace.

Banks pay well in Canada? Which ones? TD and CIBC both gave me crap offers.

I doubt any "best of us" stay in Canada. It is a fiercely mediocre place.

The US is trying very hard to convince the best of us to move back to Canada.

As well as making some of the natives wish they could go with (but that's pretty much been the case since at least late 60s.

When I was deciding on where to immigrate to in 2006, I had job offers from the UK, Canada and the USA.

I ruled out the USA because of the CFAA. I do not like second guessing everything I do online whether it violates a badly written federal law. Of course, United States v. Elcom Ltd. didn't make me much happier. Three Felonies A Day came out later and it just strengthens all this. We also should mention the almost complete lack of social net and the insane health care system.

There are numerous arguments that by now the United States is a failed state https://www.thenation.com/article/world/trump-usa-election/ https://eand.co/how-america-collapsed-and-became-a-fourth-wo... etc.

> banking, healthcare, telecom, they are all stuck somewhere in the 90s.

Some of the eHealth stuff uses active FTP connections, so this isn't even an exaggeration.

Hmm, you probably shouldn’t look at how ACH works then

A friend has one of his jobs with a telecom company. Everything there is political. He bets that the day the outage happened, as many devs as possible would have tried to call in sick as that is what happens at his firm (Telus/Bell).

I would have. I call in sick during prod failures at one of my gov jobs.

In jobs with no real rewards for excellence but decent job security, a crisis is a reason to get out the door.

> no-blame culture

We have this where I work and I have mixed feelings about it. On one hand, it's certainly freeing that we don't get fired for mistakes. On another, it's strange to feel that the people who objectively caused the headache for everyone else go without serious attention to remediation.

IMO, we don't have to name names, but Root Cause Analysis should include the decision tree that the primary motivator chose. No blame culture also hampers a lot of insider-threat analysis.

No-blame culture and no-consequence culture are not the same thing. Just because you don’t get fired doesn’t mean the org should allow someone the capability to make the same mistake, whether that’s access/permissions/change in org etc

I think the idea behind no-blame culture is that a single person shouldn't be able to cause a massive incident on their own (assuming no malice), so finding a victim to put the blame on isn't helpful anyway. Also, if you start blaming people, everyone will go in permanent cover-my-ass-mode and this will usually just lead to some poor junior, who couldn't play the game yet, getting the full blow.

Of course, letting people run rampant and acting everything is fine is not great, but there's a good chance it might be an organizational problem and, if not, a HR problem (which is not necessarily related to a specific incident).

I think the idea behind no-blame is that there are basically three ways that an outage could be the fault of an individual: maliciousness, incompetence, or bad process. In the first two cases, the individual should be fired, and in the third case, blaming the individual is counterproductive as it not only doesn't do anything to mitigate the issue but also distracts from the real cause. The only gray area I can think of is where it's not not clear from a single incident whether an individual is actually incompetent or if they're "borderline competent" (maybe more competent at other aspects of the job role and it's a balance to figure out if they're net value is worth keeping around), but in that case, I think that's also probably better to be dealt with at the management level and keep out of the root cause analysis.

> MO, we don't have to name names, but Root Cause Analysis should include the decision tree that the primary motivator chose.

I definitely agree with this! I actually have seen cases where people have asked for revisions to a root cause analysis not due to a lack of naming individuals but a lack of sufficient explanation for what the anonymous individual actually did to cause the outage, which I think is fair feedback to give.

> No blame culture also hampers a lot of insider-threat analysis.

Yeah, if no-blame culture is being used as a shield for maliciousness or incompetence, I think that could have a harmful effect on things. I guess I always just interpreted "no-blame" as actually meaning "no blame for honest mistakes", but I guess that's too long to be catchy.

If incompetent people got in the position where they cause a production outage whose fault is that? Sure they should be moved / privileges stripped -- consequences are fine -- but once again, in my opinion, aside from maliciousness an outage is always an organizational mistake.

I have twenty years of experience in the framework we work with and I can't roll a production hotfix without QA approval.

What this comes down to is: if prod goes down, I should be able to focus on getting it back up and not worrying whether I have a job tomorrow. That's not conducive to success.

Except that no one person even enormously missing something should be able to cause a major error.

no blame works without exceptions when the team makes it impossible to screw up big (test coverage, multiple reviews, modularity, canaries, DR strategy etc.)

if you do retros and the same thing keeps coming up and it's always coming from the same person, then the manager will have to solve that

If you can take down the whole network, it isn't a single person's fuck up. Its the whole organization's fuck up.

Not me. Canadian "tech" has 0 talent left. We're all working for the much better paying american mega corps

Even shopify is getting bled dry

As a Canadian who moved to the US to work in big tech, I love it down here!

Reply on news.ycombinator.com