Hacker News Reader

Comment by darkwater | Hacker News Reader

Or not. I'm remote and I cannot ask questions nor coordinate action to solve live production problems due to this outage. Slack is becoming a SPOF for many organizations, especially distributed.

View on HN

For anyone else curious, SPOF means single point of failure in this context.

I was thinking "single point of focus."

Why not both?

That's not really how acronyms work, that doesn't convey the same meaning.

At my workplace we were told SPOF was too negative, so one of the engineers changed it to SPOS. Single Point Of Success.

It makes sense to have a fallback method set ahead of time, as others have mentioned.

You won't have access to the things slack integrates with but at least you can talk.

I know some people use IRC as a low-dependency fallback channel.

we use a mumble server as backup and parallel mean to discuss in the operations department... Sometimes talking all together using voice as if we were in the same office is much nicer than writing!

Mumble / murmur [1] is great. I set one up for a family member in minutes. Super easy to deploy. People should use their RC releases. They are solid, sound quality is great and it scales really well. A small to medium VM can handle thousands of people.

[1] - https://wiki.mumble.info/wiki/Main_Page

At one stage when there was an outage about a year ago, folks at work were scrambling around trying to get HipChat resurrected (which we'd been using prior to moving to Slack).

I had an ircd of some description installed, configured and ready for use in under 3 minutes, and started to get my team using it as a stop gap measure. Had that rolling long before either the outage was resolved, or the HipChat server was resurrected. People forget just how stupidly easy it is to set up IRC.

Do you have a recommended server / client? The last time I tried to use IRC (joining an existing chat) I actually had quite a bit of trouble, and ended up just giving up.

irssi for a client

Freenode runs this one for a server.

https://github.com/freenode/ircd-seven

Inspircd is pretty easy to set up (as a server). Hexchat is a great client, Weechat is great for a commandline client. If you have server access you can also set up Kiwi which is a webclient, making it easy for those with less tech prowess to connect.

For ages, I ran unrealircd [1] and anope [2] services. One could put TheLounge [3] (web front end with redis) in front of it for persistence. Not as easy as inspire to set up, but rock solid and feature rich.

[1] - https://www.unrealircd.org/

[2] - https://www.anope.org/

[3] - https://thelounge.chat/

Having a single communication medium is a SPOF no matter what name it has

Isn't internet a single communication medium anyway?

It is, technically. Good thing there's also phones and SMS.

I have heard a horror story involving a phone line multiplexed over fiber to an IX and someone used it as OOB.

Both of which increasingly work over the Internet on the backend, from what I hear.

But it is routed. Also your laptop usually has 2 Network cards on different physical layers... Not the same.

The internet is specifically designed to route around downed links such that there is no SPOF, that’s the theory anyway.

However, when major backbones go down it takes it a while to recover.

Well, kinda! Just look at this recent CloudFlare blog post[1]. It's true, theres no Single Point of Failure - there are MANY points of failure ;)

[1]: https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-...

The internet, like the road network is designed to cope with failures - even if a major problem leads to congestion.

We build services on top of that that don't cope with failures.

Does this change anything though? The internet has many of these SPOF services. And even more AS's who through malice or fluke can take trigger a failure of those SPOFs + more.

At the end of the day, that sounds like a massive pile of SPOFs that just barely keep up..

What OSI layer are we discussing?

Interestingly that you say this; we need an additional layer called layer 8.

As app is 7 but we have grown beyond the seven into meta layers of service types which have become the favric of how we operate... such as email, txt, chat etc to keep all other aspects of infra up and running.

the OSI model is not a truthful representation of the different layer of networking. Things get messy between layer 2 and 3 (ARP for instance?).

Also, everything above layer 4 is a enormous mess, considering that stuff is usually happening at the application itself anyways. There are network protocols which have proper OSI seperation, but they are rarely used. (IS-IS in combination with CLNS for instance).

Knowledge atrophy is a serious thing... i used to be deeply skilled in all of that but attention is where the heart is... so ive literally forgotten so much abt networking over the years.

A failure of mine was to not keep a project portfolio runbook which i could look back upon.

Imagine a system where all your status reports were logged and searchable - that would be move valuable than linkedin...

Not really? The internet is literally an internet, i.e. a network that connects networks. You can nearly always find another connection that still works, be it a coffee shop or cellular.

Sure, having a single connection is a SPOF, but anyone living in an urban area and anyone with a cellphone (and not using the same cellular carrier for home internet) has more than one connection.

I'm on Google Fi, so theoretically I have redundancy on my phone as well. I have used my phone as a hotspot for work, and I've also gone to other locations to get work done (library, coffee shop, a neighboring company's lobby, etc).

A lot has to go wrong for me to not get work done. If Slack goes down, I use text, email, or Jira comments (in roughly that order of priority). Relying on only one communications method is foolhardy.

It depends.

I had a similar outage at another job for a day. Pre slack days, but instant messaging and email was down.

I needed to do some things so I just stopped checking with people. So did other people.

The result was I found was that we were double checking, coordinating and doing a lot of verification that honestly wasn't needed. The sky didn't fall, everyone still saw the changes and were ok with it.

After that folks stopped doing a lot of the traditional coordination that proved to be superfluous and maybe never accomplished anything. The handful of mistakes that happened, were also easily caught / fixed.

Granted, this requires people to make good choices.

Good thing email still exists, right?

Same here. I'm on support duty and have my own outage to attend to, and that's really really hard when you cannot communicate with your experts.

Forgive me, but just to clarify... There is no “fallback” communication channel for your team to use?

There’s email, phone, whats app, group SMS, some form of WebEx or Zoom, or even a shared google doc spreadsheet with tasks for each outage.

If Slack is your only forum of communication with your team it’s time to rethink your support structure and DR plans.

Edit: It’s very unhelpful if I point out issues and don’t provide solutions. PagerDuty has a great article about Incident Response[1].

[1]https://response.pagerduty.com/

I don't want to sound snarky by questioning what's probably an org mandated tool usage, but I can't quite believe companies where there's only one accepted way to get in contact.

Ideal would be a live conference call (phone, Skype, Hangouts etc...), but you've got SMS, email, heck, I once solved a pressing issue with my team in a Sharepoint Word document where all our comments were just new paragraphs being edited live (which actually had the handy benefit of being savable without needing to pay extra for the privilege...).

I know it's 'old school', but we would all just get on a conference call before the fancy chat tools.

Yeah, i understand. hosting your own mattermost (or other slack clone) is not enough for you? You can also have 2 self-hosted solution on different servers (one on our own, one on any CSP), in case your server fails.

Why can't you use email?

That is rather unfortunate if there is only a single channel that an organization can use in a production outage situation.

Well, there are other means of communication but they either:

* do not fit well for a quick message interchange like you can have during an outage (email)

* don't have the whole company (or at least the whole tech team) on it (whatsapp, telegram, you name it)

* make compliated to share big chunks of text/logs (phone call)

It probably makes sense creating another secondary - usually dormant - communication hub like a WhatsApp group for emergencies but people tend to misuse it (like using it to ping for some minor incident when all the other more suited communication tools are working perfectly).

Anyway it wasn't a big outage on our systems, just a minor hiccup, but it's a lesson learnt nonetheless.

Do phones exist?

This is part of the problem - we all need to ask too many questions. We should be working on pre-defined tasks most of the time, where it doesn't matter if it's waterfall pre-defined or scrum time allocated.

The commenter above said they had to "solve live production problems", how exactly would "pre-defined tasks" help them?

I slipped over the "live" word, but I still stand by what I said. I most probably work in a similar environment, with tonnes of live production issues, and I believe it's a result of not being allowed to proactively plan for situations - most of the issues are a result of cost savings, systems driven to their absolute maximum capacity, etc. Live production issues are a byproduct of moving too fast, breaking too many things.

I'd style my way out of that argument by saying out reliance on tools like Slack rather than other pre-planned work builds a reliance on that one tool for all tasks being done "in the moment", which then means it's relied on as the medium in live production issues.

If there was a lower reliance on IM for all tasks, other solutions for crisis management would be utilised (potentially).

What an incredibly arrogant generalization to make. You know there are also production issues created by moving too slow and failing to address pending scalability issues, right?

heh.

> failing to address pending scalability issues

How is that not related to not getting enough funding and resources?

And yes, having scalability issues means moving too fast - the _business_ moving too fast.

This is starting to sound like the business version of starting physics problems with “imagine a frictionless surface...”

Reply on news.ycombinator.com