How we scaled Slack to support 1000s of developers
blog.railway.com
blog.railway.com
Does it mean that it forces the customers to send 50 messages instead of a single mail message?
> However, developers hate email.
Always feels awkward when there are general statements about the group you belong to, which are certainly false at least for you, and possibly for most others you know in that group.
If anything, I would expect developers to dislike a web chat, as tech-savvy people generally seem to prefer more lightweight and customizable options.
> That context helps Railway Engineers deliver a truly first class support experience.
Much of the post reads like a rather annoying advertisement.
I was curious why it was a problem to support thousands of customers in the first place, and looks like the issues were in combining conversations over different communication channels, and that a single overloaded person used to create those Slack chats.
It's not perfect, for sure, but:
1. You can use a client of your choice.
2. You can filter in any way you like.
3. You're not stuck with any particular provider, and migration is not a big deal.
4. You can organise things in a way that works for you (inbox zero etc.), I find that a lot easier than the typical notifications.
5. It comes with all kinds of "integrations", e.g. GitLab comments can just be responded to via email, which I use a ton when on the go. A CI failure, customer question etc you can just forward to discuss it. When set up reasonably, it's the one communication channel you need.
It takes some discipline and self organisation, but I haven't heard about developers hating email. I think there's a ton of reasons not to hate it. Is it non-trivial to use well? Sure. Do others use it in questionable ways? Sure. The same is true for Slack and all that as well. But I can't think of anything that I found easier to manage at the end of the day.
The worst part is that pretty much every popular client spews massive amounts of unnecessary HTML and CSS into the body.
What I don't have yet is the same for my bulk import to SQL database, but I'll probably figure something out in the next project where we need to analyse large amounts of email.
The one thing I like better about Slack over email is the lack of constant spam from random apps.
Working in a company means that only 1% of my email is actually from humans. I must constantly add new filters, because even small companies today use dozens of SaaS products that are constantly fighting for my attention, and only a small fraction of those messages is actually useful for me.
Slack so far is just humans. Plus integrations mostly added by developers for developers.
Of course this can change at any time, and I'll have random robot spam in Slack. Which means I will have no option other than start ignoring Slack as well and start asking people to come to my desk. Which might cause me to finally go into carpentry as I have been promising to do for the last 15 years.
But I would warn you: Slack doesn't solve the "stealing your attention issue" that you are currently experiencing only with email.
They didn't program the bot very well though, because it had a separate employee->acct list from the org that you could modify to change who was messaged.
The problem with email is that you subscribe to a service (or “get subscribed to”) and suddenly there’s 200 messages per week on your inbox that you gotta filter out and figure what the heck you’re gonna do with…
The worst part is that you often NEED some of those notifications, a small portion of them at least, but an email notification is the shittiest form of notification ever when SaaS apps are involved, and SaaS apps rarely allow for good configuration granularity.
Due to many people doing this, all important notifications that are actually actionable are now sent via Slack as well. So far these notifications are only for things where I actually need to urgently respond or do something, so none of the spam has yet reached Slack.
Also, no one ever seems to personally send an email to another employee because Slack is always the better option, and people actually respond on Slack, unlike via email.
If there’s one thing I hate more than email, it’s not being able to mark an item as unread, and put it in a folder to address it later. Slack is synonym of stress.
The Slack stress compounds with being pinged all the time, on various topics. It transforms a developer role into a manager role.
The pings are within your control. If your org has an unruly Slack workspace, get on top of your notification settings or just leave it closed most of the time and check in periodically.
Here's a good intro https://news.ycombinator.com/item?id=15747703
Also, the 2 minute rule itself is flexible, just like the rest of the system. Sometimes it's a 30 second rule, and sometimes it's a 5 minute rule.
If someone wants to raise issues from marginal sources, let them go through the friction of messaging in the main channel or DMing.
You don't need to react to everything.
I'll zoom out- Railway got its first users from a Twitter post from the founder asking people to try the product. The founder then sat in a Discord voice channel during the winter break just answering mundane questions from people. Railway (like many DevTools at the time in 2021) was significantly more free than what it was today.
When I joined, we made the determination that Support Engineering, for the sake of our customers (staying sharp) and mental stimulation (we're not just help desk) - build our own automations to help users. (Blog post here: https://blog.railway.com/p/support-engineering-is-engineerin...)
(Railway also, intentionally, keeps our team small. Not out of elitism but out of alignment. The hardest part of working remote is making sure that people are on the same page. The tradeoff is that we are busier and need to be mindful about what work we take on. I, however, take it all on!)
Around 18 months ago or so- I started getting pulled into more revenue conversations as we moved upmarket. Railway moved into it's first proper plans. A lot of the prevailing advice we were getting was to start gating the product and start booking demos. So we did that for a bit much to the annoyance to developers. Then we started emailing them to see if they wanted to chat- I can't speak for you, but a lot of people in our book didn't reply. (Blog post: https://blog.railway.com/p/scale-not-sales) In Railway's eyes- we think that Support/Sales is just the same thing just at different points of the developer's journey. A lot of engagements begin with a question.
The channel that did work was Slack Connect. After answering a question or two with companies on there- their usage would grow a lot. So we scaled that out!
Again, sorry if it read like an ad- not my intent.
This looks like an amusing misunderstanding in both directions: I did not check the linked previous blog post until now, and interpreted it as developers disliking to use email to contact you, but apparently what you mean is that you contact them, for no important reason. I would not just ignore such an email, but probably would treat it as spam, and this sounds like the kind of thing some people dislike email for.
My perspective is every form of communication bears certain unspoken rules / implications.
* Email - This is a serious thought out discussion
* Slack / Similar in a shared channel - This is office-formal, you should be professional but can joke around. Similar to how you would talk to an equal or your manager in person
* Slack / similar in DM - This is a rapid "Quick question" forum where you just need some links or quick thoughts. Nothing here is very serious
* Phone call - This is informal, rapid discussion for time sensitive issues. I'm calling you so that I don't have to page you.
At least I can organize and search emails by default, and it's relatively private.
It's annoying having to anonymize my issues for my employer so I can publicly request support.
I get that it's a preference thing, so I'm not saying my position is correct for you. If you don't care about any of these things, I can see Slack being better.
"Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that"."
I had experience from the customer side of this situation, when Segment set up a Slack channel for an integration I was doing a few years ago. Lots of back and forth with their team (that would probably have been better as email), with some good, important stuff about how to work around particular issues.
Then one day, I get a message from a Success Engineer of some description saying that now we're up and running, they're going to "Archive" the channel. No worries. That must mean it'll go read-only, but at least I'll still have access to that good info in there. Better go copy/paste the whole thread somewhere safe though, just in case.
No. "Archive" is Slack for "Destroy Permanently". By the time I got the email, the Slack channel was 404ing. No more record of a conversation having ever taken place.
Yikes. I still haven't gotten over "Customer Success Agent".
I am so not ready for Success Engineering.
AI, take me away.
You wrote this and then;
>would expect developers to dislike a web chat, as tech-savvy people generally seem to prefer more lightweight
What?
Web chats are not heavy
[0] https://beuke.org/terminal-latency/#:~:text=6.2-,firefox%20(... [1] https://danluu.com/term-latency/#:~:text=terminal.app%20and%... [2] https://pavelfatin.com/typing-with-pleasure/#:~:text=0.8-,Em...
- Endless channel creation that buries crucial info...
- Confusing side-panel threads that are easily missed, leading to fragmented discussions...
- Notification overload that bombards you with pings...
It's success it's a mystery, I can only attribute to fashion or the hype of chasing the latest trend.
Have you used Microsoft Teams?
- Has much richer message formatting and doesn't regularly do weird things to mess up the formatting as I'm composing a message. (Just this morning I was fighting Slack to try have nested bullet points not disappearing or de-indenting as I was composing a message).
- Consistently marks messages as read without me having to click chats multiple times for the unread indicator to disappear. With Slack I somewhat regularly have this problem, more than once a week.
- Like every other application on Windows, Ctrl+K works for adding a hyperlink. Slack is in fact so aware that people are used to using the Ctrl+K shortcut it even catches you trying to do it and then tells you how to add hyperlinks in their own special snowflake way.
- Doesn't clutter my Windows notification centre. Unless I clicked the notification center message it doesn't ever disappear. WhatsApp by comparison which also uses the Windows notification center does this perfectly. Slack is the same on iOS, it doesn't clear messages from the notification center once a message is read unless you clicked the notification.
Teams has its warts for sure, but I find it's considerably more polished than Slack in general.
Windows 11 Pro on my Dell Latitude 5540 and it's a rock solid experience.
I on principle always format my disk drive when getting a new PC and reinstall using an ISO downloaded from Microsoft to ensure no OEM software is on them. For my Dell the only thing I install is "Dell Command | Update" (not to be confused with Dell SupportAssist which is crap targeting general consumers). It's for updating drivers and firmware and it does absolutely no nonsense, only prompting me when a driver or firmware update is available.
I use no third-party anti-malware, no "corporate management software", just Windows Defender, but with Dev Drive and exclusions to my working directories so everything is very fast for my development work with Visual Studio.
My work is mostly developing for .NET and Electron which works on Linux and macOS as well. WSL with Visual Studio is super impressive that I can seamlessly debug processes running under WSL from normal Visual Studio. For Electron / NodeJS debugging I can use VS Code which also supports debugging processes running under WSL.
I have done two registry tweaks to Windows 11 to remove a couple of annoyances:
- I've turned off the "simplified" context menu which shows by default in Windows Explorer, otherwise most of what I need is hidden behind an extra click of my mouse.
- I've turned off internet search results from showing on the Start Menu. This is a huge quality of life improvement, I've never ever cared for searching outside of my web browser and it makes the Start Menu super responsive and stops it showing web results above stuff on my local computer which was what I was looking for.
(As a side remark, I was very happy when I only somewhat recently worked out how to get iOS to stop showing web search results when I search my home screen for apps, that was a huge annoyance as I only want to search for apps on my phone by name and often it would show web searches over a locally installed app name.)
People constantly complain here about all the adverts on Windows 11. I honestly don't get any adverts, maybe it's because I'm in South Africa and not the US (marketing sounds like it's beyond obnoxious there). I think it came pre-installed (even when installing with the ISO downloaded from MS) with a few rubbish Windows Store apps (like Candy Crush), but I uninstalled them normally and nothing ever came back. I also use OneDrive already (which integrates really nicely with Windows), maybe if I didn't have it set up, Windows would be bugging me about it.
Really, everything just works seamlessly for me.
Install NetLimiter and be amazed with the MB/s of data that is getting uploaded into Microsoft when your PC is at rest, even when you open PowerPoint and are just editing a slide..
Install SystemInformer (the successor of ProcessHacker) and be amazed how Microsoft Telemetry will use 12 to 15% of your CPU even with the machine at rest...
This is with me running:
- Several browser windows
- Multiple Visual Studio instances (with ReSharper)
- Several Electron(-like) based Applications
- Outlook Classic
- Slack
- Teams
- Docker Desktop
As is plain from the screenshot, there is not MB/s of network traffic, furthermore, the CPU is also pretty idle as one would expect from my not doing anything heavy (like compilation) at this moment. This is very typical for my PC most of the time.
Based on my decades of using Windows full time and my screenshot, your comment is absolutely not representative of typical idle resource usage.
I do however acknowledge that there is a recurring scheduled telemetry collection process which when it runs pegs an entire CPU Core for minutes. I do find this annoying when I notice it, but that's maybe every couple of months which isn't enough of an annoyance to me to waste my time worrying about it, or I might have actually already disabled it in Task Scheduler.
Some people on principle dislike telemetry which is a completely understandable viewpoint, but for me personally I'm okay with Microsoft collecting essentially anonymous usage statistics to better understand how people use their software and where best to direct their efforts.
Notifications off by default so people create new channels with you as a member, write extremely important information inside them and you find out weeks later.
Each post is like an announcement so nobody uses them for the everyday trivial stuff you need a channel for. For casual technical discussion, asking for help. Whenever you post in a channel, assuming people enabled notifications and your post will be actually seen, everyone is compelled to answer as the UI screams for attention.
And don't get me started about it hiding part of a post by default so you answer thinking you read everything but you missed an essential part because post was 4 rows and 2 were hidden.
I also use Discord privately, so I know the alternative
https://answers.microsoft.com/en-us/msteams/forum/all/in-my-...
Unable to pause screen during a presentation, one of the most basic features of a professional platform. A feature requested here by more than 2460 users, for at least the last five years:
https://feedbackportal.microsoft.com/feedback/idea/a5063e83-...
Microsoft is fast-food not a Michelin restaurant...
I wish Slack had a "don't mute this channel completely, but only show it as unread once a day".
95% of people that say this don't mute channels (at all or enough).
I worked at a startup where our single biggest cost was the Slack bill. It's crazy to me what companies will go through to "seem professional"
- free voice/video channels (persistent meeting rooms)
- put text/voice channels into categories
- RBAC for channel categories, text and voice channels (only engineering role can see engineering stuff)
- mute notifications from specific channels or categories
probably some more but can’t think of them off the top of my head.
straight up, if i were starting something tomorrow, i’d use discord and not slack.
Incredibly frustrating to search for something, see that people have discussed and answered it before, but not be able to read it.
- Tech debt, they can't easily add or change features anymore
- Performance, rendering syntax highlighted markdown in messages is too expensive
- Performance optimization, by not having syntax highlighting they encourage people to create snippets instead, which goes to different backends and lazily loaded clientside rendering pipelines to support larger files and syntax highlighting.
It's this. I've worked on Slack applications and you can clearly see why they might struggle to implement this given the data structure of their messages. Their brand of markup is also quirky and it might be difficult to add this feature in a backwards compatible way
Snippets was added much later and part of me feels like the demand for syntax highlighting was exactly why they added snippets. It's also a scapegoat for them not adding syntax highlighting to regular messaging
You can desync the WYSWIG nature of it to the point where you submit text that was "invisible" to you while also potentially deleting text in the submission that you can still see.
A more simple to reproduce bug is that you can't edit before the first character of a syntax block. The cursor skips from outside the syntax block to after the first character inside the block.
I've not actually used it in a professional setting but there are a large number of very active/thriving coding related communities (mostly open source, but not exclusively) and Discord seems to handle them well.
That exact point was my very axe to grind and why I am thankful to work on all sorts of comms. automations at Railway when they let me.
Developer of the feature here, happy to chat through details of the implementation and general developer GTM watercooler advice for those who are building for developers.
What are your SLOs for messages?
How are you managing missed SLOs?
Railway has three support engineers who work on tooling + myself if you count me. (I have been more focused on customer migrations and classical sales as of late.)
One thing that I think we left out of the blog post which would have been helpful context is in our support tool, we have a queue of emails, Discord threads, forum posts, tickets, and Slack threads. Depending on the customer plan, we have an SLO assigned to the conversation. Enterprise people get a response in 1 hour, Pro in 24, and Hobby is best effort (usually 2-3 days at worst).
If we hit any of those timelines, the support on-call (a rotating person who is on the operations/GTM side of the house we call logistics) will get a page on PagerDuty. If the page is missed, it escalates up. We do a on-call rotation for support namely to make sure that we build the empathy for our customers for everyone at Railway while also testing if we built good tools that people know how to use. It's one thing to build something for yourself, a whole other thing to build for others.
However, what's interesting is there is a direct correlation to the amount of conversations we can support and churn! We did put the walls up I'd say 18 months ago. Although it resulted in an okay business outcome as today, user churn numbers actually increased controlling for pricing changes.
Ever since we invested in our support tooling, we've been able to field more conversations and I think we (and the users) are better for it.
Talking to users- no matter how "small" they may be is a potential relationship for us to invest in. We will try to keep our heads to the ground always listening to users.
Talking to users and using our own product are the two hills we will die on. A company dies when they stop doing both, long before any financial reckoning I feel.
The blog post is a bit old but the point mostly stands- we try to keep our customers close to us, but lately, we have extolled the virtues of focus work, so we rotate the customer comms baton through our on-call system.
I prefer email. Email is asynchronous, It does not interrupt my flow or my presentations,
I can decide when I spend tine responding. Then I can fully focus on the emails and type the response they merit.
Of course you have the person sending you an email and 30s later they tap you on the shoulder to ask if you got that email.
The more I think about this, the more "don't shoot the messenger" fallacies I see around me.
Like, the complaints about overly bureaucratic documentation was interpreted as critique of documentation by "people". Etc.
Sometimes it even works...
No VNCs/Virtual desktops.
Why is that? If i'm paying for resource usage and i don't see what's illegal about VNC, why is that?
It genuinely sucks that to stop 90 percent of suspicious use cases, you do get legitimate use in that net. For that I apologize.
Now since Railway has a better handle on fraud, we feel that we can loosen up the restrictions, so if a customer clarifies their use case, we don't mind. With that said, we don't offer raw VMs (yet) so you will find the platform a bit limiting a bit after our complete migration to self-hosted metal.
After saying that, let me share that Railway is incredible and very affordable.
Hypothesizing aside, there are some interesting options out there like Tightknit and (for Discord) AnswerOverflow. I still think that Slack makes it way too hard to spin up and start up a community- I think there are many Slacks like the CNCF's one that would be nice as a browsable index of content.
As such, Railway's been investing in our own forums for this reason http://help.railway.com/ because we've noticed the decline of forum culture - and we want to bring it back. (Async, well thought out responses)
Maybe this is the next step of that integration!
https://slack.com/marketplace/A044USEAHRV-unblocked
I will try it out and see how it performs.
However, a lot of my time is trying to reach companies where they are and the majority of them are in Slack. I think there is a case to be made where more companies use Matrix over Discord for their community operations but I think we are seeing a trend of users willing to forego freedoms (as in speech) in exchange for services that are free (as in beer).
I think the challenge for protocol developers is to make sure that their client experience is top notch to win and keep those users, something that Mastodon faced vs. the now popular rise of Bluesky. The policy our company practices on whether to support a channel is: we build where our users are.
With that said, it's looking like a Teams integration is more likely so I guess we will be learning how to hook into there...
(Rip Spectrum chat)
I don't mind email, but I despise Slack.
I miss the olden days of being able to check my email once or twice a day with nobody expecting an immediate response, and being able to focus the rest of the time.
Three or four times a year, in response to something genuinely urgent, I might get a phone call.