Slack Incident Resolved – Degraded service affecting multiple features | Hacker News Reader

Slack Incident Resolved – Degraded service affecting multiple features

status.slack.com

84 points7ewis49 comments

AWS Status page[1] shows a couple of issues in eu-west-2.

Amazon Elastic Compute Cloud (London)

Instance Connectivity

3:21 AM PDT We are investigating instance connectivity issues in a single Availability Zone (euw2-az2) in the EU-WEST-2 Region.

Amazon Relational Database Service (London)

Small number of instances unavailable in a single Availability Zone

3:36 AM PDT We are investigating connectivity issues affecting some instances in a single Availability Zone (euw2-az2) in the EU-WEST-2 Region.

[1] https://status.aws.amazon.com/

I don't quite understand this. Aren't the availability zones denoted by a letter?

My Aurora instances are in eu-west-2a/eu-west-2b/eu-west-2c, so what is euw2-az2?

The letters are different per-account.

thanks, first time seeing that

It's so the load is evenly distributed across AZs because I bet most people just select -az-1.

As another commenters mentioned, the AZ letter different per account.

https://docs.aws.amazon.com/ram/latest/userguide/working-wit...

“To identify the location of your resources relative to your accounts, you must use the AZ ID, which is a unique and consistent identifier for an Availability Zone. For example, use1-az1 is an AZ ID for the us-east-1 Region and it is the same location in every AWS account”

So each separate AWS account will have a different AZ name that maps to use1-az1, but use1-az1 is a region-wide constant.

I'm frustrated that AWS managed services are not moving away from this AZ. I'm seeing CloudFront returning a 504 trying to connect to API Gateway for a number of requests. After 2 hours you would think they would stop trying to route traffic there.

What are good current open source (self-hostable) alternatives? I have the following:

[1] https://rocket.chat/ (Works well, sadly the mobile client is no longer native and it shows)

[2] https://matrix.org/ (More IM than workgroup chat)

[3] https://mattermost.com/

https://zulip.com/

Just curious what makes you feel Matrix is more IM than workgroup chat? What's missing or different to be suitable for you?

I've actually heard several people have the inverse sentiment ("Matrix is more group chat and channels than IM")

Matrix is a protocol - you’re probably thinking of https://element.io (formerly Riot.im)? If anything, Element is more workgroup than IM (but there are other Matrix clients which are more IM than workgroup).

The funny thing about theirs website is cookie notice. Either they didn't read GDPR (which is unlikely), because you don't have to ask consent for "We use cookies to avoid showing you duplicate content, ..." or they're trying to get sneaky.

You have to obtain consent if your pass data to third parties but again, what is the point of advertising privacy on the page then.

You wouldn't see the cookie's consent on the apple website.

Presuming the banner on this https://element.io/

I had the same understanding as you but if you read https://gdpr.eu/cookies/ , you may come away differently. Specifically, with regards to "strictly necessary cookies" like these, it says:

> While it is not required to obtain consent for these cookies, what they do and why they are necessary should be explained to the user.

The page is just an overview however and it doesn't state where you need to explain to users what the "strictly necessary" cookies do.

Here is the thing:

- consent should not be required and user should be able to decline; Could you see this options?

- check browser storage, they set you cookies when you open a site, before any click on their's consent banner

It's not a consent banner. It doesn't ask you for consent, it informs you what cookies are used for.

My bad. I have developed banner blindness to a stage when I see every banner as a consent popup.

Still don't understand why they have decided to annoy users with this thing.

Sites are allowed to set cookies without consent when they are strictly necessary for site functionality. They do not need to provide you with the option to decline. The gray area is that they need explain why and how the cookies are strictly necessary but the prescriptions around that explanation are missing.

XMPP cries in a dark corner...

Matrix also can perfectly work as an alternative for slack. You can have rooms that are private to your server, you can add integrations for bots and so on...

The lesson of Matrix is that you have to give everybody all of the features. All at once. In advance. Even the ones nerds don't need. Then they'll be there for everyone else.

Already tried Movim https://movim.eu/ :) ?

I did! I logged in and then I cried in a corner, alone and with no one to interact with.

IRC.

https://flock.com

Doesn't appear to be open source

https://zulipchat.com/

Probably the best I have used, easily prefer it to Slack but it's hard to convince people to use stuff other than Slack.

I'm just not a fan of React Native mobile apps (same with rocket chat). Specifically Android, they are large and slow.

I recently looked into an online banks APK (also some JS framework) and there was one 100k line JS file with hundreds of lines of "if IOS" in it, on android...

I have no problem with any chat apps written in React Native. I've used Discord's mobile app and haven't heard of any complaints from anyone using it and the 4.5 to 5 stars from its users on iOS and Android show this.

Their Electron desktop apps however, just like the rest of the chat clients from Slack, Element, Keybase, Zulip, Rocket.Chat are a complete horror show. Feasting on my poor MacBook's RAM and disk space like an all-you-can-eat-for-free buffet until the disk and RAM is full. Waste of resources for a chat app on the desktop.

With Mac Catalyst, there is no excuse for this carnage on macOS. Other solutions are less mature but I would rather have React Native maturing on the desktop for Mac, Windows and Linux as an alternative to Electron.

I second zulip, it offers most of the features you want, is truly self hosted so your data stays home (very important for me in my business) and the self hosted version is not missing half the features (eg full search).

Only thing missing is a more modern UI (I like the UI as is but in non tech companies it's hard for many users to use it compared to the cutesy competitors).

The stream / topic system works great once you get into it, but make sure to use it rather than fight against it.

We self host Mattermost and it works well. Logs are in Postgres which is nice. Building the React Native app was not super fun but we have some experience in that already which helped.

It's a good alternative for teams unwilling to shell out every month for a Slack subscription, if you're willing to put in the work to get it setup.

Does the prebuilt app not allow you to point it towards another server, or why did you have to build it yourself?

If you want to get push notifications from an in-house server, you need to have your own FCM project to host the push notifications. In that case, you need to compile the app yourself.

We've used Mattermost for the past few years with great success. Currently looking at moving into Teams - however stuck with no migration path for data into Teams (can export OK out of MM though).

Will be a shame to see MM go

https://bitrix24.com is pretty good as Slack alternative as well. Free in cloud for unlimited users, open source edition for on-premises hosting is available as well, but it's $$$.

EU west 2 AWS has been having issues this morning. We've had a load of outages ourselves, they could be experiencing something similar.

Yup, we've had Aurora issues that are ongoing there.

From London, Slack was resolving to a load balancer in eu-west-2 earlier during the outage. It now resolves to eu-west-1 and I have no issues connecting to Slack.

  $ nslookup edgeapi.slack.com
  Server:  1.1.1.1
  Address: 1.1.1.1#53
  
  Non-authoritative answer:
  edgeapi.slack.com canonical name = prod-haproxy-edge-nlb-eip-038c466f971aa8b8.elb.eu-west-2.amazonaws.com.
  Name: prod-haproxy-edge-nlb-eip-038c466f971aa8b8.elb.eu-west-2.amazonaws.com
  Address: 35.176.156.206

  $ nslookup edgeapi.slack.com
  Server:  1.1.1.1
  Address: 1.1.1.1#53
  
  Non-authoritative answer:
  edgeapi.slack.com canonical name = prod-haproxy-edge-nlb-eip-1bffe8d4fbdf4b94.elb.eu-west-1.amazonaws.com.
  Name: prod-haproxy-edge-nlb-eip-1bffe8d4fbdf4b94.elb.eu-west-1.amazonaws.com
  Address: 18.200.77.145

We just saw a flurry of IoT devices losing connectivity to eu-west-2 as well, all seems to have recovered now though.

Back to back with the Zoom outage no less.

And the gmail attachment outage.

Was Zoom outage yesterday related to this? AWS has become sort of a single point of failure

Why doesn't Slack fall over to another AZ?

They did eventually, the ingress point was originally eu-west-2 but got changed to eu-west-1

I thought Zoom had moved to Oracle cloud.

Images on our site haven’t been loading for customers most of the day, related to this.

This doesn’t explain why slack has been generally terrible most of August though.

Humble guess, some people don't work during holidays

All most instances in London are down

Anyone else thinks slack web version is very slow ?

Don't Put All your Eggs in One Basket (AWS)

Reply on news.ycombinator.com