Slack – Degraded service affecting multiple features | Hacker News Reader

Slack – Degraded service affecting multiple features

status.slack.com

249 pointskimi235 comments

Just yesterday I was musing that if I were King of the (World|Company) I'd want an open-source Slack-alike that I could just drop into the Cloud of my choice and operate entirely within my private network, subject to my own access control just like other internal services, and with full access to all message histories in whatever database-like thing it uses in its Cloud. Sure, I'd still have a SPOF but it's game over anyway if my Cloud goes dark.

Is there such a project, and if so does it have any traction in the real world?

(Oh and ditto for every damn thing Atlassian owns. Yikes.)

https://mattermost.com/

We use this at my company - perfectly reasonable UI, don't know about the APIs/integrations, which I assume are way behind Slack...

The Mattermost API is a superset of Slack's. "Most" integrations "just work", the rest need a minimum of tweaking to DTRT.

Mattermost API being a superset of Slack's seems to be mostly true, but looking at the features our Slack app uses I don't see block kit (https://api.slack.com/block-kit) or message scheduling. If we were to integrate with Mattermost it doesn't look like it'd be a ton of work though

Have to say Mattermost's API docs are so much easier to read than Slack's.

Slack's API documentation and handling of accounts/bots is so obscure. I've never seen anything that bad from a major company.

Dunno about feature parity but zulip is great.

https://zulipchat.com

https://github.com/zulip/zulip

Zulip, Rocket.Chat, and Mattermost are probably the best options.

Best thing is that some messages go through despite returning 500s. This leads to our integrations spamming our companies channels with redundant messages while retrying. :/

Ouch, that one hurts. I don't know slack API but the contract expectation in the call should absolutely be "500 means the message failed to send"

Tells me there's some subsystem throwing the error, not the actual messaging (or, if intermittent, something wrong with the distribution.) What do you want to bet it's some kind of new analytics?

It's not possible to always ensure that you report whether an operation succeeded. Consider a timeout - you waited 5 seconds for the backend to respond and heard nothing - was it, or will it be successful? You don't know.

Typically a 500 should be considered as a "may or may not have succeeded", and steps taken to deduplicate operations if necessary.

Http 500 is like when the server drops connection before returning a response. Something happened, but you have no idea what happened. You can't safely retry unless your request is idempotent enough.

Is it just me, or do those outages seem far more common lately? Or maybe this is happening just as often as it used to, but now we're growing more dependent on cloud-based software so this is reported more prominently?

More dependant.

both - have a look at Slack's status history. They seem to have an outage of some sort every month

Can someone explain to me why any company should outsource something as critical as internal communication to a company that is 5 years old? The video messaging left aside, is it really that better than time-tested solutions like email or just running an IRC server?

no, it's not. It's just a company based whatsapp group chat you can use while looking busy

It has emojis and is the Next Best Thing (tm)

Because they don’t trust their IT or operations groups to manage IRC or any other open source communication system; even as they trust those same people to run the software that makes them money.

/shrug

IRC is just a protocol, neither the server nor the client do have to be open source.

It's a reasonably nice client, and available on every platform.

It has easy integrations with basically everything, and non-developers can set them up on their own.

It has bouncer functionality built in - You don't miss messages. It can also archive them, and handle them in a corporate-compliant way without needing to speed a bunch of time setting that up manually.

Because it's a central company and not IRC push-notifications are easy.

Emoji are shared corporate wide, anyone can add them.

It's a really nice out of the box experience.

You can absolutely replicate it with enough setup and tweaking, but the free version of Slack will go a long way.

"just running an IRC server" is... a lot. Or can be. Who is running it? Does it scale well? How is login synced with corporate logins? What about mobile devices?

It's not "better" than email, it's different. Real time. It has a million and one service integrations, which IRC doesn't.

Point is, it's really not all that crazy to just pay a third party to manage all this stuff for you. You might imagine the alternative is a perfect IRC server with 100% uptime, but I doubt that would be the reality. If you're saying a service like this is too important to be run by a third party then you're going to have to invest a lot internally to come up with an alternative that's more reliable.

> Does it scale well?

It's IRC. On modern hardware. It's practically infinite.

> it has a million and one service integrations, which IRC doesn't.

There and were are things way before Slack, eg https://loqi.me/

Obviously you can create IRC integrations for anything. But how many are installable with one click, like Slack integrations are? Having to write custom implementations is yet another extra task and point of failure compared to just paying someone to manage it all for you.

another "but".

First "but it doesn't scale". Now "but integrations".

There will always be another "but", and there will always be a link on the internet showing that it's been done already.

"but shiny web gui" - https://thelounge.chat/

"but mobile app" - https://github.com/MCMrARM/revolution-irc

"but it's complicated" - https://www.mirc.com/ since 1995

IRC does have limits. Some. One is that to make it "modern" it needs a bouncer - that said znc is spectacularly simple to set up with it's web interface.

Integrations were never one of those limits. Just google IRC bots.

How do you make it possible for people to view the IRC channel history when they were disconnected? I know you can have a bot that saves a log, but can it be made seamless?

I think it's really sad how the world has migrated away from an open protocol to a closed one, but I remember IRC having some very real annoyances, well beyond stuff like integrations.

The answer is in at the bottom of my previous entry. It's called an IRC bouncer.

bouncer, or web client that persists chats, like TheLounge or Convos.

> another "but"

That's my entire point. It's death by a thousand cuts. Now I'm responsible for an IRC server, a web frontend, a mobile app and a bouncer.

The OP said "just run an IRC server". There is no "Slack or just an IRC server". There's "Slack or a number of connected, interdependent services you need to maintain". I'm not saying it can't be done, or that it isn't the right answer for some people, but it's also a lot more work (and thus, expense) than just paying for Slack. It absolutely makes sense that people do it.

Why is paying another company better than getting employees and paying them the same or even less money?!

When did this madness start?

In what world is it going to be "the same or even less money" to hire one or two sysadmins on a full-time basis than paying for a Slack subscription?

I don't know how large your company is, but with 10k+ employees, it's suddenly a reasonable option.

With that many employees you're signing up for Slack's enterprise tier, and you can bet they price it so that they come out the right side of this cost/benefit calculation for a lot of companies.

I have seen those numbers. You could easily afford paying sysadmins with those numbers.

But managing an IRC server cluster for 10.000+ users is no simple task.

Paying experts in their field (chat tools in this case) is most of the time the most reliable and cheapest variant. You must also include that Slack has plenty of features that are not available in IRC and mean the productivity is not as good.

Since when is running any service for a number of people something people compare with running google?

30 years ago, efnet was (apparently) 35k users.

Tbh., if you need two full-time sysadmins to run your IRC server + bouncer + log backup + web frontend, you're doing something really wrong. It's more like an after-school job for a bored CS student.

Features is not the only thing, does it have well funded sales team and strong corporate support ? Slack itself did support IRC till recently, the product is not just a sum of its features

> does it have well funded sales team and strong corporate support

Get 2 sysadmins. 2 because of redundancy. Pay them to maintain your service.

"well funded sales team and corporate support". F that. Trust your employees.

2 sysadmins will cost considerably more than a few slack accounts. Unless your company is huge.

> Unless your company is huge.

Yes. Someone above have mentioned ibm using slack though. I think that's huge enough for the argument.

Yes at that scale it may be useful to roll out a custom system. However I'd count into account that large customers probably have different pricing and that maintaining custom systems is always bothersome because you are always the largest customer.

"Well funded sales team" is unfortunately what makes closed products win over open ones.

What happens when your sysadmins say "Maintaining IRC is a waste of time and gets us 1% of the features Slack does, let's just get Slack?" (At my last company, it was the sysadmin team, including myself, who decided to switch to Slack and killed our IRC server.)

Do you stop trusting them then?

Come on, in the 90s and early 00s, volunteers ran and managed IRC networks scaling up to hundreds of thousands of users. This was back when hardware and bandwidth were much weaker and 100x more expensive. It's really not that hard.

Yes, volunteers ran these things. How many? Because you're going to need to have paid employees rather than volunteers running your servers. Is it going to be more reliable and cost effective than just paying a third party to run a service for you?

For something like IRC? Yes. The service is dead simple and one server can handle thousands of users with modest hardware and without breaking a sweat. A completely unfamiliar sysadmin could set it up in an hour or less. Since it's probably internal, there's little need for public services (nick/channel registration, etc).

So what about the mobile device issue? Corporate logins?

If you want to match what Slack offers you aren't just running an IRC server, you're going to have a small fleet of dependent services that, added up, just about match what Slack does.

Is it really that surprising that people just opt to pay for someone to deal with all of this for them?

At which point you're not looking at IRC as a service anymore. There are different niches served - but the question was about the reasonableness of setting up IRC alone, which is still dead easy.

Put TheLounge [1] in front of it for people that want a web interface. Perhaps use ldap for corp users. [2]

[1] - https://thelounge.chat/

[2] - https://thelounge.chat/docs/configuration#ldap-support

Most of how I use Slack, and how everyone I know uses Slack, involves 1. sending messages to people who are “away” to get their attention/loop them into a conversation; and 2. people reading back up in history for context when they come online.

Now, modern IRCds might have features I’ve never experienced on raw IRC... but if they don’t, I don’t suppose you’re suggesting that this server also run thousands of bouncer bots in order to provide this contextual history feature?

(Also, whence push notifications?)

If you're a big enough company (more than 60 people if computer are your means of production, around a thousand if not), you should have your own server and a fallback on a CSP. Just pay some guy in India/South Africa to manage all your internal services and architecture. If they are sensible and you need to trust the guy, just pay him well and hire locally.

I can set you up a resillient architecture and 3 services (communication, documentation, filesharing) in less than two weeks (let's say two weeks and a half because security concerns/limitations and all that stuff). Add one week and you will have mail server, VOIP communication, log conservation and monitoring, backup images on CEPH, vnc service for "easy" maintainance and a small documentation explaining the few action needed to maintain the monster for two years. Initial cost for a company: 15 to 25k$, + 1 hour and ~100$ a month.

Not necessarily better than IRC, but email is a pain.

They somehow got Clevels on the bandwagon. Those who actually understand the implications don't have a word.

Truth is I see many capable people buying into these as well because of the convenience - read: lazyness - factor. The "ain' nobody got time for that" effects everyone and the willingness to pay for services had skyrocketed in the past decade. This applies to blogging, to home servers, backup, anything. We're all guilty at some level.

IRC is pretty much useless since you don't have history. Comparing Slack and IRC are like comparing TCP and HTML5. Why IRC is pretty bad in enterprise:

- no audio / video

- no history

- no SSO / AD integration

- no rich features like links, pictures, upload docs ect ...

- easy bot integration with APIs, I can have Datadog pushing things to Slack ( alerts, graph... )

- no archiving channels on the go, Slack is very useful when you have an incident and you create a temp channel and then archive it when the incident is closed.

matrix does all that and can be self hosted - ours is.

Does the opensource matrix server scale? I may be wrong but I got the impression it was more an example implementation and was not used by Matrix in production?

There is no closed source Matrix server implementation that I'm aware of. Synapse is the server used in production for matrix.org, the French government deployments, and tons of other servers including my own. It is somewhat resource intensive but it can scale using workers.

There is of course several other server implementations in progress but none are fully ready yet.

IRC is a protocol.

The features you're listing is server/client dependent.

> no audio / video

There are products that are far better for audio and video communication than Slack. At my company, we use Zoom.

> no history

Can be done with a bouncer if you need it.

> no rich features like links, pictures, upload docs ect

Is there a reason why an external service can't be used? I can't upload a image here on Hacker News, but I can always include a link to it.

> easy bot integration with APIs

So I have to use the HTTP protocol to manage a bot rather than checking for certain key words in the message itself.

> no archiving channels on the go, Slack is very useful when you have an incident and you create a temp channel and then archive it when the incident is closed.

That depends on the settings. One thing that happens every so often at work is that people get trapped as the last remaining person in a channel because only an administrator can archive a channel. In IRC, that's not an issue.

Ive always said Slack is IRC for millennials.

"The Wheel of Time turns, and Ages come and pass, leaving memories that become legend. Legend fades to myth, and even myth is long forgotten when the Age that gave it birth comes again."

I certainly prefer it, but I've both heard many other millennials strongly oppose slack, and Gen X:ers talk highly about other "no-setup" systems. I don't think not a generational thing.

Plenty of millennials grew up with IRC.

* search engine that just work

* push-notifications that just work

* client available for enough people

* copy / pasting images that just work (the'll never admit it, but animated gif is their killer feature)

* stable-enough infrastructure for something that costs you 15€/month/user not taken from your "salary" budget (often a different and substantially less taxed part)

* and their outages are funny (receiving messages twice is arguably better than not receiving message)

Yes, it's because it "just kinda works for what it costs, and you can forget about it most of the time and do your job."

This seems to be characterized as "laziness" by some people, a point I honestly don't understand (I though lazyness was a virtue in our circles.)

That being said, there are legitimate reasons to not use it:

* software licence

* data property

* technical implementation of the native clients

* general distrust of centralized solutions

* cost of having IM at all (see Newport's whole publishing career)

At least, it means there's an opportunity to create a company that handles decentralized user-friendly open-source standard-based data-respectful attention-preserving uber-reliable chat system. HN users would be a market.

> search engine that just work

For some definitions of "just" and "working". Even ignoring the business limitation, the UX of Slack search is meh. That said, I'm yet to see any of the modern web&mobile-oriented IMs that would have a decent search UX.

But yeah, your list is solid. They offer plenty of value, just mixed with plenty of negatives.

A big source of the complaints against Slack is usually the network effect - in context of both work and OSS communities, it's usually somebody else that makes the decision and imposes it on you, forcing you to run face-first into all the "legitimate reasons to not use it" you listed. Half of the pain would be gone if they opened up their service to third-party clients; I'd say the reason they didn't have as much opposition from people initially is because they run the bait-and-switch with IRC gateway.

Interesting point about : work environment vs communities.

For work, the decision of which IM tool is most likely going to be a corporate one, without much choice of solution. I guess the problem here is the same as for other corporate tools. (And people have strong feelings about corporate tools, but it seems there are less strong feelings about "using jira vs bugzilla" as there are "using slack vs irc". Or maybe there are as much strong feelings.)

On the other end, I get the troubles with the "network" effect for out-of-work communities (open source projects, clubs, etc...) Such communities can at least have "some" level of debate about which solution to use, and balance the pros and cons as they see fit. It's completely acceptable for an OSS community to discourage using such platforms - as any other non-free, centralized platforms. We're still not forced to use anything.

Then, there is the "club of non-power user" situation. I would be hard pressed to ask someone from my drama club to setup and maintain an IRC server (neither do I want to do it.)

Thanks for adding this dimension.

Follow-up: is there a "nice" provider of some slack-like solution (mattermost, etc...) that hosts servers for communities of non-techies for not too much money ?

What does years have to do with it? Look how many engineers have to sit on their hands when github goes down?

- Mobile client, including mobile push

- Search

- Scrollback/history available in the main client interface people who newly join a channel (extraordinarily important during incident response)

- Ability to link to a message in history

- Ability to handle multi-line messages in a vaguely competent matter

- Image uploads, snippets, etc.

- Users of a GUI tool / web interface are first-class, users of a command-line client are second-class; this is unlike IRC, and important when the goal is a single chat system for the entire company, not just engineering (and usually a subset of engineering at that)

- Client/UI expectation that you do not catch up on every message unless you want to (like IRC, but unlike email)

- Quality integrations that have already been written with the third-party apps that you're already using

IRC and email have both been time-tested, and have failed the test.

Aside from features:

I have seen enough corporate mail servers going down, irc and jabber servers breaking etc.

Using those won't magically give you 24/7 with more than five nines uptime.

Key difference is: If slack goes down many companies are affected and it goes on the internet. If a company's mail setup goes down employees are a bit annoyed, but after a while it works again and nobody outside notices and it's no news on HN (unless it's a real major thing bringing a notable company completely down)

Our company switched from self-hosted RocketChat to Slack just this week. Personally I didn't agree with it, but it's not my call. The reason was mostly that Op guys were too busy with other things and didn't want to care about yet another critical company-wide service. They though outsourcing it would solve the downtimes and bugs. This outage couldn't come at a better time.

I’ve heard from countless of friends how many times Outlook goes down for their company, whether it’s a midsize company or a large company. I don’t think this is unique to Slack or company age.

We used to run an IRC server at work, and it was pretty exclusionary to non-engineers; setting up something that'd log backscroll required maintaining a vm somewhere, searching through history required another service, etc. If all slack does is chat (and video communication by way of an acquisiton), I'm happy letting them be 5 years old. I'd scrutinize a company that: 1. has no fallback mechanism of Slack is mission-critical and goes offline 2. isn't careful about including non-technical stakeholders in communications.

Edit: clarifying that I'm talking about my workplace, not a company I run.

One problem with Slack we have in the finance world is that _literally everything_ employees say on company assets needs to be recorded and readily accessible for audit (No this is not an invasion of privacy, you're on company time on company systems, we don't peer into your personal phone or anything).

Unfortunately, Slack does not do this very well. I'd really like to know if there's another service that _does_ do this well.

Zulip and a very big disk.

Looked at this, appears to make it difficult by default: https://zulip.readthedocs.io/en/latest/production/security-m...

My guess is most of the major chat players handle this pretty easily -- it's a non-starter for many potential users if you don't. Zulip stores all data indefinitely (including message edits, etc) in its default configuration.

Mattermost, but you'll have to manage it yourself.

Symphony chat, what goldmans is trying to aim as the bloomberg IB killer. It's quite nice and designed for banks and API/Bot friendly.

url:https://symphony.com/en-US

Genuine question - when I use slack - I'm able to see messages going back to when the channel was created for my organization as well as all files shared by date. I'm not sure how slack doesn't do this well?

i would guess Microsoft Teams can do this.

Slack records everything you do in an immuteable audit log. It's used by lots of finance companies as well as government institutions ( who have just as strict if not stricter rules ). Also they have lots of features to help with e-discovery, so not only is everything recorded, but getting your needed data out is quite easy.

Probably Teams. https://docs.microsoft.com/en-us/microsoftteams/security-com...

We used (and I assume they continue to use, hi COFers!) Slack at Capital One. They are super careful about compliance so I'm sure everything was above board with the regulators.

Mattermost does, and just raised about $70m to do it well. ING is on their website. They have e-discovery tools and it's all hosted on your own cloud/servers.

There are reasons why email is async and supposed to be decentralized, with priority based fallback options. (The priority in mx records). Most email servers try to deliver to fallbacks, and even if that fails, will try to deliver for _days_.

For all people who think slack can replace email, think about these safeguards.

Isn't it better in nearly every situation to know something has failed immediately, rather than get a delivery failure notice a couple of days later?

In general mail servers will reply back with a failure notice within an hour or so. This isn't a standardized behavior and timeouts and retry semantics differ.

The whole point of email is fire and forget

If it's a temporary outage, eg. network loss of a datacenter, email is not lost. It will be delivered with a delay, tried over and over again, until the timeout limit is reached. Due to the async nature the delay is not a problem.

I don't think any serious IM platform really intended to replace email, they serve different (but related) purposes.

- https://www.nytimes.com/2019/06/19/style/slack-replace-email...

- https://www.brianmcadam.com/5-reasons-replace-email-text-im-...

- https://tech.co/news/slack-replacing-email-workplace-2018-08

[...]

It is Friday evening somewhere!

it is Friday evening in South Asia

If it's any consolation, their status board has actually been reporting errors (hi cloudflare)

You're implying we don't update the status board when there are errors?

The recent outage was very very delayed. I stand by my comment.

Also don't get me wrong, you do a great service. It's just a pet peeve that it seems invariably status pages are a lie.

Let me quote from the previous HN discussion

”"”sauldcosta 4 days ago [-]

We use downdetector.com because status pages tend to take up to an hour or so to update, if they ever do. reply

jgrahamc 4 days ago [-]

1042 UTC First alert of global traffic problem 1057 UTC Internal group chat room up and running 1102 UTC Status page updated So, first alert to status page was 20 minutes.”””

In those 20m we had repeatedly checked your status page, realised it was our issue and started pulling engineers to deal with it as per procedure. People are on call, it's highly disruptive.

Surely you knew within those 20m that something was up?

In the end we realised it wasn't our issue because we checked Twitter.

Edit - added quotes

Yep. I tend to agree that we could have gone faster. The difficulty is that getting clear information out fast when you are dealing with a difficult situation is hard.

But, I guess we could have put some status up quicker.

Thanks for the acknowledgement. Appreciate it's hard, false positives etc. Glad you see our PoV.

I do.

I'm irritated, however, that you picked on Cloudflare as a bad actor here when we strive to be transparent and quick to get out full information whereas others (e.g. Amazon) are slow as mud.

But I get that being on the other side of this is difficult when you don't have information.

Well, it's understandable, if OP has been burned by Cloudflare's slowness in the past, and not by Amazon's.

Amazon, Google, seem far far worse. I wasn't intending for my post to me representative, just that the CF outage is recent and affects me more personally than the others atm. Also, I had zero expectation my comment would actually be read, this is quite a surprise.

If it's any incentive, if the status page had any inkling that something was wrong, I'd be first on here posting "omg their status page is real".

Fair enough. Thanks for making the comment. Made me think about how fast we act in terms of public status pages. And sorry that route leak affected you.

Cloudflare seems to have decent status reporting. I'd go with AWS on this.

No outage for me, but I'm seeing some messages appearing twice. Is that related?

> Is that related ?

That's the symptom of the outage as far as I can tell.

The replies to this status tweet seem to indicate it’s related

https://twitter.com/slackstatus/status/1144577107759996928?s...

My org is having a few people see that, or messages getting silently dropped, or failing to edit an existing message.

Okay, yeah. I'm seeing some of these issues now.

It's odd because the message said it failed to edit, but my client still shows the message as edited.

I'm getting a bunch of duplicate messages. Luckily these guys are building a chat platform and not financial systems.

If slack outage brings it on top of Hacker News, I think it's a compliment!

In other news: an unexplained productivity spike has been recorded today across the tech industry.

Or not. I'm remote and I cannot ask questions nor coordinate action to solve live production problems due to this outage. Slack is becoming a SPOF for many organizations, especially distributed.

For anyone else curious, SPOF means single point of failure in this context.

I was thinking "single point of focus."

Why not both?

That's not really how acronyms work, that doesn't convey the same meaning.

At my workplace we were told SPOF was too negative, so one of the engineers changed it to SPOS. Single Point Of Success.

Same here. I'm on support duty and have my own outage to attend to, and that's really really hard when you cannot communicate with your experts.

Forgive me, but just to clarify... There is no “fallback” communication channel for your team to use?

There’s email, phone, whats app, group SMS, some form of WebEx or Zoom, or even a shared google doc spreadsheet with tasks for each outage.

If Slack is your only forum of communication with your team it’s time to rethink your support structure and DR plans.

Edit: It’s very unhelpful if I point out issues and don’t provide solutions. PagerDuty has a great article about Incident Response[1].

[1]https://response.pagerduty.com/

I don't want to sound snarky by questioning what's probably an org mandated tool usage, but I can't quite believe companies where there's only one accepted way to get in contact.

Ideal would be a live conference call (phone, Skype, Hangouts etc...), but you've got SMS, email, heck, I once solved a pressing issue with my team in a Sharepoint Word document where all our comments were just new paragraphs being edited live (which actually had the handy benefit of being savable without needing to pay extra for the privilege...).

I know it's 'old school', but we would all just get on a conference call before the fancy chat tools.

Having a single communication medium is a SPOF no matter what name it has

Isn't internet a single communication medium anyway?

It is, technically. Good thing there's also phones and SMS.

I have heard a horror story involving a phone line multiplexed over fiber to an IX and someone used it as OOB.

Both of which increasingly work over the Internet on the backend, from what I hear.

But it is routed. Also your laptop usually has 2 Network cards on different physical layers... Not the same.

The internet is specifically designed to route around downed links such that there is no SPOF, that’s the theory anyway.

However, when major backbones go down it takes it a while to recover.

Well, kinda! Just look at this recent CloudFlare blog post[1]. It's true, theres no Single Point of Failure - there are MANY points of failure ;)

[1]: https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-...

The internet, like the road network is designed to cope with failures - even if a major problem leads to congestion.

We build services on top of that that don't cope with failures.

Does this change anything though? The internet has many of these SPOF services. And even more AS's who through malice or fluke can take trigger a failure of those SPOFs + more.

At the end of the day, that sounds like a massive pile of SPOFs that just barely keep up..

What OSI layer are we discussing?

Interestingly that you say this; we need an additional layer called layer 8.

As app is 7 but we have grown beyond the seven into meta layers of service types which have become the favric of how we operate... such as email, txt, chat etc to keep all other aspects of infra up and running.

the OSI model is not a truthful representation of the different layer of networking. Things get messy between layer 2 and 3 (ARP for instance?).

Also, everything above layer 4 is a enormous mess, considering that stuff is usually happening at the application itself anyways. There are network protocols which have proper OSI seperation, but they are rarely used. (IS-IS in combination with CLNS for instance).

Knowledge atrophy is a serious thing... i used to be deeply skilled in all of that but attention is where the heart is... so ive literally forgotten so much abt networking over the years.

A failure of mine was to not keep a project portfolio runbook which i could look back upon.

Imagine a system where all your status reports were logged and searchable - that would be move valuable than linkedin...

Not really? The internet is literally an internet, i.e. a network that connects networks. You can nearly always find another connection that still works, be it a coffee shop or cellular.

Sure, having a single connection is a SPOF, but anyone living in an urban area and anyone with a cellphone (and not using the same cellular carrier for home internet) has more than one connection.

I'm on Google Fi, so theoretically I have redundancy on my phone as well. I have used my phone as a hotspot for work, and I've also gone to other locations to get work done (library, coffee shop, a neighboring company's lobby, etc).

A lot has to go wrong for me to not get work done. If Slack goes down, I use text, email, or Jira comments (in roughly that order of priority). Relying on only one communications method is foolhardy.

This is part of the problem - we all need to ask too many questions. We should be working on pre-defined tasks most of the time, where it doesn't matter if it's waterfall pre-defined or scrum time allocated.

The commenter above said they had to "solve live production problems", how exactly would "pre-defined tasks" help them?

I slipped over the "live" word, but I still stand by what I said. I most probably work in a similar environment, with tonnes of live production issues, and I believe it's a result of not being allowed to proactively plan for situations - most of the issues are a result of cost savings, systems driven to their absolute maximum capacity, etc. Live production issues are a byproduct of moving too fast, breaking too many things.

I'd style my way out of that argument by saying out reliance on tools like Slack rather than other pre-planned work builds a reliance on that one tool for all tasks being done "in the moment", which then means it's relied on as the medium in live production issues.

If there was a lower reliance on IM for all tasks, other solutions for crisis management would be utilised (potentially).

What an incredibly arrogant generalization to make. You know there are also production issues created by moving too slow and failing to address pending scalability issues, right?

heh.

> failing to address pending scalability issues

How is that not related to not getting enough funding and resources?

And yes, having scalability issues means moving too fast - the _business_ moving too fast.

This is starting to sound like the business version of starting physics problems with “imagine a frictionless surface...”

That is rather unfortunate if there is only a single channel that an organization can use in a production outage situation.

Well, there are other means of communication but they either:

* do not fit well for a quick message interchange like you can have during an outage (email)

* don't have the whole company (or at least the whole tech team) on it (whatsapp, telegram, you name it)

* make compliated to share big chunks of text/logs (phone call)

It probably makes sense creating another secondary - usually dormant - communication hub like a WhatsApp group for emergencies but people tend to misuse it (like using it to ping for some minor incident when all the other more suited communication tools are working perfectly).

Anyway it wasn't a big outage on our systems, just a minor hiccup, but it's a lesson learnt nonetheless.

Why can't you use email?

Good thing email still exists, right?

Do phones exist?

It makes sense to have a fallback method set ahead of time, as others have mentioned.

You won't have access to the things slack integrates with but at least you can talk.

I know some people use IRC as a low-dependency fallback channel.

we use a mumble server as backup and parallel mean to discuss in the operations department... Sometimes talking all together using voice as if we were in the same office is much nicer than writing!

Mumble / murmur [1] is great. I set one up for a family member in minutes. Super easy to deploy. People should use their RC releases. They are solid, sound quality is great and it scales really well. A small to medium VM can handle thousands of people.

[1] - https://wiki.mumble.info/wiki/Main_Page

At one stage when there was an outage about a year ago, folks at work were scrambling around trying to get HipChat resurrected (which we'd been using prior to moving to Slack).

I had an ircd of some description installed, configured and ready for use in under 3 minutes, and started to get my team using it as a stop gap measure. Had that rolling long before either the outage was resolved, or the HipChat server was resurrected. People forget just how stupidly easy it is to set up IRC.

Do you have a recommended server / client? The last time I tried to use IRC (joining an existing chat) I actually had quite a bit of trouble, and ended up just giving up.

irssi for a client

Freenode runs this one for a server.

https://github.com/freenode/ircd-seven

Inspircd is pretty easy to set up (as a server). Hexchat is a great client, Weechat is great for a commandline client. If you have server access you can also set up Kiwi which is a webclient, making it easy for those with less tech prowess to connect.

For ages, I ran unrealircd [1] and anope [2] services. One could put TheLounge [3] (web front end with redis) in front of it for persistence. Not as easy as inspire to set up, but rock solid and feature rich.

[1] - https://www.unrealircd.org/

[2] - https://www.anope.org/

[3] - https://thelounge.chat/

It depends.

I had a similar outage at another job for a day. Pre slack days, but instant messaging and email was down.

I needed to do some things so I just stopped checking with people. So did other people.

The result was I found was that we were double checking, coordinating and doing a lot of verification that honestly wasn't needed. The sky didn't fall, everyone still saw the changes and were ok with it.

After that folks stopped doing a lot of the traditional coordination that proved to be superfluous and maybe never accomplished anything. The handful of mistakes that happened, were also easily caught / fixed.

Granted, this requires people to make good choices.

Yeah, i understand. hosting your own mattermost (or other slack clone) is not enough for you? You can also have 2 self-hosted solution on different servers (one on our own, one on any CSP), in case your server fails.

According to some prospective studies, some people would (and please stay with me on that) actually use slack to communicate about work-related matters with co-workers.

More study are needed to clarify if this work-related communication might actually be required for work to get done. (Cal Newport is definitely preparing a book on that any time soon.)

Unfortunately, the researchers are too busy maintaining their IRC server to actual do it - but at least they're using IRC.

Whereas, my productivity definitely goes up when HN is down ;)

I’ve hosted multiple IRC servers over the years and can’t remember ever having to do “maintaining” after the initial setup.

Honestly curious, how long did you last without having to:

* update the version of the IRC server

* update the version of the os of the machine running the IRC server

* repair a broken disk / fan / overflowed disk on the machine

* add / remove / reset password / change weird settings of users

I'm not saying this is "unbearable", and plenty of organisations have people whose job description would probably correspond to doing those tasks.

But I can definitely understand why you would want to skip them entirely and have someone host your chat - which is basically the job slack is paid for.

"Mission critical, but not paid by your customer" is always tricky to staff for, isn't it ?

>* update the version of the IRC server

Odds are that you’ll have updates worth installing once every couple of years.

>* update the version of the os of the machine running the IRC server

Almost never unless there’s a remotely exploitable code execution vulnerability. Local bugs wont matter unless you run multiple services on the same box.

>* repair a broken disk / fan / overflowed disk on the machine

Depends on your hosting setup. With a cloud setup perhaps never.

I’m certainly not trying to suggest that anyone should use IRC over slack in any situation, just that it’s not a horrible time sink requiring significant maintenance.

I think what may have triggered people here were the absolute terms (can’t remember ever having to do “maintaining” after the initial setup).

In this forum you'll be hard pressed to find someone who thinks it's a good idea to leave a cloud-based (or any internet-facing, really) server completely without maintenance for any significant period of time.

If you had said that usually it's as simple as checking in every now and again to reboot / make sure security updates are installed / check functionality and that it's a manageable burden which is worth the effort you might have gotten a more favourable response.

Edit: or if you had automated it to an acceptable level, maybe some more information about that.

But it says slack right on the box

That is a remarkably good point that I can't believe I hadn't noticed. What a strange thing to name a product that one would try to sell as Enterprise productivity software.

I wonder if this will also link to a dip in RAM sales.

I'm part of a mostly remote team. Slack going down sucks. And Pre-slack suuuuuuuuucked.

You mean I have to open OpsGenie and Grafana now instead of waiting for a ping in the ops channel???

Also, how many IT service desk ticket systems just lit up with people asking them to fix Slack?

Has the CTO and key technical staff retired after the Slack IPO ? :-) In the UK there is no real outage, although Slack struggles sending some messages and this can result in some messages appearing twice. This problem seems to be worse when communicating with offsite staff, onsite messaging seems to be fine.

They should call this sort of event a 'Slackening'

You must not be on the $12/user/month plan.

By accident I noticed that you can add as many characters as you want to the end of the url and it will still load that page. Neat.

Came here for the "Slack shouldn't exist anyway" comments and was not disappointed. Or rather I was, but in a different way.

Best wishes to all of the folks at Slack and those who depend on it.

Honestly, the same. It's tiring that people just has to suggest self-hosting IRC or any other software all the time when Slack is mentioned. It's a good way to communicate and it's easy. Hope they resolve it, overall a bad situation.

So now everybody can get back to $WORK

Wooo productivity!!

A resurgence of IRC is nigh!

reminds me of that one time a genius at IBM created an `ibm-global-announcements` channel and force-invited 200'000 people in it, and then some guy `@channel`d and all ibm's slack workspaces were down for 30 minutes

https://status.slack.com/ibm/2018-03/f01d4c22cd953dd7

https://i.imgur.com/Rk6Kdgp.png

EDIT:

also that channel made using slack impossible for mac book air users, they had around 80% cpu usage for slack. so basically entire marketing and PM part was unable to work that day. developes machines were wasting around 20% on slack.

after people started complaining in that channel, posting in it was limited to admins only, but they didn't lock commenting. so, for approx 6 hours all of IBM was posting memes in ＴＨＥＴＨＲＥＡＤ as we dubbed it, and @mentioning the genius who created that channel. next day the channel was nuked, not even an archive preserved.

some guy calculated that the entire affair, considering electricity prices, a 20% decrease in developers productivity, and so on resulted in IBM loosing several millions with that stunt

fun times

p.s. shout out to Martinj for that spicy jeff-coffee-mug meme

There should be a way to turn off @here and @channel. It should be a user-level setting, a channel-level setting, and an organization-level setting. Even if my org or channel doesn't opt out, I should be able to.

I hate that it can't be controlled per channel. There are many channels where it's useful and desirable, so our org doesn't disable it globally. This creates situations where new people @here and @channel not knowing any better, which leads to hurt feelings all around.

Some of this you can already do. Users can opt out of @here and @channel for any given channel. I know that orgs can restrict usage of @everyone, I assume they can do the same for @here/@channel but I'm not sure.

I know you can in Discord, I'd be shocked if Slack doesn't let you do that. checks Settings

Yes, you can mute a channel and suppress @everyone and @here if you want. Still friggin nuts that a chat service can bring a modern machine to it's knees.

Ah I knew you could mute it but muting a channel does not suppress @here. I did more exploring and found it's a multi-step process to suppress @here vs muting, but it is indeed possible.

Learn something new every day.

Or at least a way to force users to pay a fee whenever they use those features.

It would be interesting to have a karma function whereby it was based on your use of @channel or @everyone - or something similar to how HN used to require 500 karma to be able to downvote...

People at my company use @channel constantly for the most trivial of things its a major pet peeve for me

My work has a 1000+ person channel for ad-hoc support questions that aren't critical and could be answered by anyone. If it's critical, our support team has an SLA, but Slack doesn't.

There was one specific person who would ask a question, expect an immediate response, and if they didn't get the response they would @here. They were instructed not to do that because it pings over 1000 people, so the next time they asked twice in an hour and when they didn't get a response they said "I hate to do this but I really need a response @here". They were kicked from the channel. Really unfortunate since it's a very useful channel to be in, but @here and @channel is just so disruptive in big channels.

It should just be turned off for large channels. Pop an error message and direct users who need it to a support article, if they follow up real support people then see how many people complain.

I remember that day fun times. I just shut off Slack and got back to work was nice not being pinged constantly.

Distributed Denial Of...Slack?

> a genius at IBM

Are you saying it is the user's fault where it is clearly Slack's inadequate implementation that is the problem?

i stand by my opinion that "lets create one chat room and invite hundreds of thousands of people into it" is a bad idea regardless of what technology implements the chat room

I respect your opinion, because even if an application could handle 200k users at once was able to be run on a MBP without serious issue, it's still gauche to forcibly invite 200k people in your company to a chat.

Slack is designed for business chat, and many businesses have thousands or more employees. Slack should be able to handle any features it provides, and @everyone is a feature. I don't know how they've implemented things or what the specific issue is, but it should at least have blocks in place that prevent stupid things. Maybe channel sizes should be limited to 1000 people or whatever their testing shows it can handle. Idk.

Software designed for large numbers of people shouldn't go down because one of those people does something stupid.

If you want to tell everyone in your company something, you could email them I guess. Is it the fact that it's a "chat room" which means you expect it not to work? What's best, here? SMS? Email? Putting something on the website and making everyone check the website once a day at noon? Tell one in ten people and get them to "pass it on"?

Is this a serious question? Companies have been using email for internal announcements for decades.

Yes, and he's making the point that it manages to work. And that perhaps your expectation should be the same for a chat client as for an email client. Which may or may not be a valid point.

And without proper mailing-list-moderation, that too can cause email storms. No one blames "the genius who decided to put everyone in one mailing list" when that happens.

The real fail here isn't putting everyone in one channel. The real fail is not using/providing proper admin tools

I'm going to guess you could also break IBM's phone system by attempting to start a conference call with 200K people on it, and destroy their auditorium by putting too many people on the stage.

We definitely blame the genius that didn't BCC an email with hundreds of individual emails (when a mailing list isn't used)

Although usually only a limited number of people can post to an "announce-list" that goes to everyone in the company. (And if there are company-wide lists that anyone can post to, most users filter those to a folder. I suspect such lists don't scale past a certain point either.)

Most companies today have a decent intranet site for these things.

I love that companies will make it hard to expense a $50 meal but they will gladly let you book an hour meeting with 20 high-paid developers or even let you directly ping 200,000 people which must cost the company many thousands of dollars.

Well, you know what happens when you email @yourcompany.com? Is it a protocol problem? Implementation problem? User problem?

It's a lot of problems at once...

I'm more surprised that the IBM server has 200k users?

According to Wikipedia, IBM has 380k employees. Not that surprising that half of the use the company Slack.

IRC works when you do that still :)

I still hate that anyone can add anyone to a channel - it's not an "invite", it's just global "add to room" privileges.

In general I find Slack not very concerned with your own ability to control your experience (no ignore user feature, can be added to rooms, can be added to rooms that then get pinged with @channel and your notifications start blowing up, etc).

This may be 'expected' in a small company where of course everyone should talk to everyone, but for larger companies and the plethora of various open-source or interest-based communities now using Slack, it's not so good.

Even worse than the channel mechanic is the thread mechanic - anyone can turn your comment to a thread, and you cannot mute threads by default. Even worse, updates to threads suppress channel unread notifications, causing you to miss updates.

Manually having to follow threads and unfollow threads is a nightmare compared to the one time configure experience of channels, especially since you cannot just mute all threads, and is easily the worst part of Slack to me.

OT, but:

> 200'000

Interesting. Why an apostrophe?

i've picked up this kind of thousand separation at some point to make large numbers scannable. c.f.

    1600000000
    1'600'000'000

separating at 10³ is due to counting customs of the region i live in.

using an apostrophe `U+0027` appears to be the least wrong use of possible glyphs accessible on my keyboard layout

also c++14 does it this way

It's common practice to use commas for this in anglophone countries, but it collisions with french decimal separator; so it's a reasonable choice.

This is not old news. There is still an IBM slack channel called #general with 30k+ people in it. And people regularly arrive in it to say "hi". No useful information is shared.

Did they enable "replace email" feature?

Why don't people use free software instead, e.g. a Matrix homeserver run via https://github.com/matrix-org/synapse/ ? IRC without a BNC doesn't really work for a corporation due to disconnects.

How well does Matrix handle VoIP?

Because it's way harder to get it running right.

Meta - can we not post outages to HN?

I understand that post mortem's are interesting learning lessons, but if you're curious whether S3 or whatever is down or not right at this moment, then use the subsequent status pages of those services.

No, this is major and should be on top of HN IMO.

HN is a more reliable status page than most status pages.

I didn't even realize it was down until I saw the post here. Just thought Slack was a little quite on a Friday morning.

Unless Hacker News is down, in which case you need to give up and go outside.

Personally, I enjoy the speculation and discussion during an outage like this. Also, occasionally someone from the mentioned company pops up in these threads

Agreed. It's fascinating to see speculation and it's almost a game when the real reason is found - we see who got it right!

It's also important - many here use Slack, and an outage may be confusing - most of us don't immediately look at status pages, so we have somewhere to go and discuss the happening.

If we could trust status pages, sure. But I've lost count of the number of times I've tried to use a service, failed, the status page says "all OK!" but Twitter and HN are full of outage reports.

Besides, the discussion here is useful. Someone has already pointed out that some messages are getting through despite returning 500s, so integrations are spamming duplicate messages everywhere. That's useful to me if I'm running an integration.

(besides x2, it's one thread with a very self explanatory title. Maybe just don't read it?)

Absolutely not. There are even times when engineers from the company will comment on HN.

If you don’t think the article is appropriate flag and move on.

FWIW, status pages for medium to large companies rarely reflect the actual status of a service. They’re powered by PMs, not monitors.

I get most of my outage notifications from HN. If it's a blip, it probably won't make the front page through my standard refresh cycle. If it persists through two refresh cycles, there's a TITSUP somewhere and it's time to take a look.

This community is awesome. Always someone with their finger on the pulse :)

Why? These events are often great opportunities to learn about system failures and their impact on other systems in real time. Status pages are often not even genuinely accurate.

Well i m not affected. But then again i dont use it

Reply on news.ycombinator.com