Signal apps DDoS'ed their own server
twitter.com
twitter.com
Circuit breaking, exponential backoff, bulkheading should be key pillars and no dev should be allowed to write a client without these key skills.
Edit: Please add Jitter to the list of items. Any good library like https://resilience4j.readme.io/ will give you all of this for free out of box!
“Bulkheading” is making sure that failures in one area don’t cascade into causing failures elsewhere analogous to how bulkheads on a ship prevent a single breach from sinking the whole ship.
While 508s were being sent, there was also a significant number of 503s (IIRC). A 503 is an absolute hallmark of something reaching max utilization. Sometimes it's a bad code push that results in memory bloat and then swap or significantly increased request handling time on a poorly threaded server (think a code push that makes a blocking linear request in a loop), but the vast majority of time an upstream dependency (specifically a data store) has been overloaded.
So for whatever reason a data store gets slower. What happens upstream when this happens? The number of incoming requests is constant, but the time each individual thread spends attempting to talk to the data store is constant (or worsening). This means each request blocks longer (resulting in potential thread starvation) or there are more concurrent requests (load) to the data store.
This creates a feedback loop of doom: as a data store slows down, its load (the number of requests it's handling at once) increases until complete failure.
The only way to stop this behavior is by "failing fast." This is how a circuit breaker works. When the data store starts responding slowly, it’s important not to hammer it with even more load, so your client watches the number of load related failures (or response times) and automatically fails requests immediately without sending it to the data store (circuit breaker). This allows the data store to become unloaded and get out of the doomed feedback loop.
"Bulkheading" is about one part not taking down a whole, which is more thoroughly described here: https://docs.microsoft.com/en-us/azure/architecture/patterns...
Even with a monolith, often one can spin up multiple copies, and assign different routes in a front end http proxy to different copies of the monolith.
It makes it extremely clear which endpoints are consuming how much resources. And if one copy goes down, the others are ok.
Absolutely one of the best things you can do, not just for protecting against faults, buy also for understanding the cost of different endpoints.
https://cloud.google.com/appengine/docs/standard/python/how-...
The idea is that at the time a request is made, you have a pretty good idea of the maximum amount of time it can take, so you attach that as metadata with the request.
If any service anywhere in infrastructure sees that a request has taken more than its budgeted time, it has permission to immediately stop handling that request and return an error.
This is all the more critical for client applications that depend on app stores to review and approve updates and end users who may not immediately update to a newer version because they don’t have auto update turned on.
Edit: I can’t seem to find it, but there was a comment recently about servers using flags to tell clients to shut up for a specific duration and avoid exacerbating problems with retries. I think it was about some specific design of Dropbox.
But if you're behind a load balancer, and the load balancer doesn't know to do that, you can't do it on the origins, because the load balancer will fail the origins and that's not good.
Just because there are selfish. a-holes who think the internet is an infinite resource intended solely for their own enjoyment, convenience and exploitation, does not mean it is okay for everyone to act that way. Not that anyone in this case is one of those types of person mind, but it's worth pointing out. This seems more like unexpected growth far outstripping provisioned resources.
Detecting if you can deliver a packet in advance is not exactly a solved problem - network detection APIs at best inform you if the PHY/MAC layer for a given connection is up, not if any given packet you generate can be routed out - and fail to detect a number of rather common edge cases (all traffic being routed over VPN on WiFi, without mobile fallback, for instance, or mobile connection appearing to be up, but not transferring data) - therefore ultimately the solution is to try, and try again, with proper backoff to prevent overwhelming your own infrastructure.
In this case the latter appears to have been missing.
This is a very hard thing to deploy after the fact and can really save your bacon, especially if you have a number of very old clients still out in the field refusing auto-updates. Sure, you can refuse to serve them, but if they get stuck in a tight loop retrying until you do, you're pretty hosed.
[0]: https://en.wikipedia.org/wiki/Thundering_herd_problem#Mitiga...
Downforeveryoneorjustme is exploring an API for service monitoring and it seems to me like every cloud client should have a standardized approach of trying to reach its own server, then checking internet access, then checking a service monitor, then checking social media status updates.
https://downforeveryoneorjustme.com/services/api
Also think apps and devices should have limited peer-to-peer information sharing instead of only talking to the operating system. In many ways our devices are like a roomful of people whispering status updates to the operating system and/or user but never talking to each other because 'security'.
Restarting an app or even manually refreshing a web page takes a lot longer than an in-memory automatic retry() function call.
To illustrate the latter option, suppose the smallest retry time unit is 1 second. The first attempt gives you a random choice in {0, 1}. The second attempt gives you a random choice in {0, 1, 2}. The third attempt gives you a random choice in {0, 1, 2, 4}. The fourth attempt gives you a random choice in {0, 1, 2, 4, 8}. This goes on until a ceiling in the number of attempts or a set wall clock time is reached.
[0]: https://en.wikipedia.org/wiki/Exponential_backoff#Example_ex...
I would tend towards increasing the minimum wait at each iteration (until you get to some maximim wait), because it it failed 10 times in the last minute, it's likely going to fail many times in the next minute, so we don't need to try more than once or twice.
Also: in case the client random is broken, you don't want to accidentaly end up with everyone retrying after zero seconds forever.
How does bulkhead helps here?
Our front end flex guy didn’t understand backing off on retries.
Some system had an issue, that caused the flex app for 1000 people to go insane. Each calling ruby app endlessly with no delay.
Ruby app remained healthy, server crashed in under 5 minutes from running out of disk space.
Our alarm interval on free space ran every 5 minutes. So no time for an alarm to sound.
A mix, of I was very proud of my little ruby app for scaling. And a a wtf, I needed to be more involved with front end people.
The native iOS mail client stopped the endless retry behavior a few releases ago (so sad it only took a decade to fix),
and so the chickens come home to roost. Moxies vehement rejection of a distributed design seems less and less tenable each outage. Last time it was what...verification numbers that werent getting sent?
and FWIW the divinations from the community are exceedingly helpful in a time when not even the signal website seems to confirm or deny any sort of outage. Signups are still being taken and the outage page at signal.org is still as static as ever. Even the twitter hasnt seen an update in nearly a day. Is anyone at Signal foundation at the controls?
As a signal user myself I know this is going to sound rude but at this point other than the endorsements from musk and dorsey, why would any new user consider this service at all if its been down for nearly two days? theres never any postmortem, and communication is generally evangelical or solicitous in nature for either installs or donations.
It's just too bad that those arguing for decentralized systems fail to spread the meme more effectively. We federalists are like cockroaches.
Signal's own official clients failed to properly back off from spamming the server with incessant requests. I'm not entirely sure how more third party clients would have helped with this issue.
Synapse (Matrix reference server implementation) had for a long time severe performance issues when dealing with large rooms. Clients would try to enter a room and would take literally forever to be able to get messages for that room.
Also, the natural reaction from people upon seeing how some rooms from bigger instances were causing problem was to create their own instances and their own rooms.
Federated protocols are anti-fragile.
Realistically, I understand that the Matrix.org homeserver is an enormous part of the network and an outage there would be comparable to this one.
However I believe that to be an orthogonal problem that also should be solved.
I'm actually curious, why does Matrix promote the Matrix.org homeserver so strongly? Why isn't there an easy link to community-run homeservers?
If you look at Mastodon for example [0], their onboarding process directs you to a registry of community servers [1], which actively promotes diversity in the network and therefore network resilience and reliability.
Matrix meanwhile, just links you to element.io, which creates an account on the matrix.org homeserver.
The web broke. We just have wait a bit while the Elders of the Internet sort this one out."
Said no one ever.
I am definitely getting strong signal that you are the type of person who has no perspective on what they are talking about, but I'm open to the idea I'm wrong and would love to be proven so. I think your expectations are out of line for the size and maturity of the company.
There was a difference in approach, though. WhatsApp was using battle-tested technology (ejabberd, XMPP) and removing things that didn't make sense to them (federation). Moxie had one trick (end-to-end encryption) and wanted to build a whole messaging platform around that. Instead of standing in shoulders of giants, he chose to ignore all the lessons from the past and do everything from scratch out of the belief that he can do better.
At some point, these systems break. In Signal's case, this point has been reached.
Technology does not exist absent its context. One might use MySQL, but is it sharded? are reads critical? Lets say you go from 1 database to 8 databases and then from 8 databases to 128? How much tooling do you think is involved in those changes (machine configuration, service discovery, shading behavior, machine replacement behavior, how machines are monitored). At what point do assumptions made under a small system completely fail in large systems? I can manually configure 8 machines by hand no problem. I can add 2 more no problem. But 100 machines? That territory starts to require automation. You can potentially look at 100 individual graphs for 100 different machines, but what about 1000 graphs? All of a sudden aggregation and drill down become necessary.
None of us (I think) have enough first hand information to understand why scaling up 10x was complicated for them. What I can almost promise is that if you have 10 servers and you go to 100, you are going to run into all types of problems you did not have at 10. Coordinated behavior across 100 machines is going to be a lot less forgiving than coordinated behavior across 10. Imagine a relationship where you have a every frontend talking to every data store. With 5 data stores, random failure isn't all that likely, but with 1000, it's much more likely. If every frontend talks to every data store, a single data store slowing down could result in ALL frontend stuck failing to make requests to it. Machine connection limits can be reached, interrupts from the NIC can bottleneck, service discovery can fail.
Does technology have an effect? Yes. Is lack of exponential backoff somewhat embarrassing? Yeah. Is failing spectacularly to keep up with exponential growth over a short period unexpected? No. I would expect any website or service already at scale to spectacularly fail to service 10x growth in a 2 week period.
By taking more that they could chew, they are burning each one of these 40M+ users. How many of these are going to keep trying other alternatives and how many are going to shrug and say "that shit doesn't work?"
In my opinion, this is a pretty unreasonable expectation. Not to mention any load testing is going to be done under a certain set of assumption, a set of assumptions that is almost guaranteed to be violated by actually scaling something.
What's the most naive load test you can think of? I am going to see the maximum amount of requests my dev instance can handle. Oh i can handle 10 requests per second. If I run 24 instances on a machine, and run 100 machines, then I must be able to service 24,000 RPS.
Ok... Now explain concretely, what kind of load testing would/should have been done that would have surfaced the lack of an exponential backoff without a priori knowledge of it?
The category of things you don't know you don't know at 10x scale is large. The justification for spending time "scale testing" is going to be a hard when your manager goes to social media and sees every single person wanting backups for their texts. "Hmm should we apply engineering resources to the feature we know everyone wants, or exploring the potential land of failures that may or may not have payoff, and definitely not immediate payoff." Most of these prioritization situations are directly and solely informed via experiencing outages, just like this one.
> then don't let it grow in the first place.
This is the type of insight you get from experience, it's also the type of call that I personally think would be very hard to make. As an engineer you don't always know if an outage is going to solve itself, take 10 minutes, 30 minutes, or take 10 hours. Then communicating that to someone with the power to create that policy...
How many (social) companies do you know of that haven't had absolutely catastrophic outages/security breeches?
This is the type of outage that justifies reliability work and gets resources applied to it. This is the type of thing that results in an organization creating a somewhat rigorous incident response policy that handles the business side (keeping users informed, giving them the best experience possible). This is a growing pain of rapid maturity.
I think you can talk the way you do because you have not personally faced this type of engineering situation (a guess). Easy to make comments from the peanut gallery, but try putting yourself in the engineer's or the CEO's shoes and walking through what information you have at the time and what options you have available at the time, without the benefit of hindsight. Imagine you have a pipeline of priorities you look at every day. Is spending time wooing a senior engineer to come work for you a lower priority than having a good story for what happens when you 10x your traffic in 2 weeks? Things can only be black and white in hindsight. Their response to this is infinitely more indicative of the type of organization they are than the actual outage.
I don't believe that it is the right way to do it. Moxie does. I rather have, work on and put skin on the game of decentralized/distributed systems. Moxie prefers to build centralized systems. He is the one putting skin on the game behind that. So I don't think we should be easy on him when the system he proposed fails so catastrophically precisely for the reasons that critics of centralized solutions said it was likely to fail.
Yeah, I would never get 40M+ users on a week. That would be by design. Instead of one company getting this many people (and potentially being killed in the process) I'd rather have 400 new providers, each working to get 10k customers. Easier to scale, more people figuring out issues, more people contributing to the system and making the infrastructure antifragile.
It appears to ultimately just front for AWS S3 backend so a very common architectural pattern.
I'm sure millions of devices doing the same thing probably bogged them down.
Even more critical since anyone can write clients that function with the server and thus a malicious actor could attach them without them knowing how to identify it or stop it.
I suppose the app that can't send messages is the most secure of all! All joking aside if their focus is security over anything else then this is acceptable. Unfortunately I don't think most of the users have the same order of priorities so it's bound to create some tension over time in multiple ways such as testing analytics, features that don't get added, or just general friction due to always requiring "most secure".
The last part I agree is probably the most concerning though, it seems like Signal's centralized services aren't ready to be battle tested from attacks bound to come due to it's popularity growth. I'm not sure matrix is the silver bullet to that problem either... email isn't resilient to DoS because it's distributed it's resilient because the key centralized players like Gmail can withstand constant attack without affecting service due to their scale. Neither Signal nor Matrix are ready for the attention that comes with serving billions from that perspective. It's something I think will come with growing pains though.
Helpful comment from this thread: https://news.ycombinator.com/item?id=25803251
This is made worse by mobile device virtualization being difficult to achieve as well. Is there a system I could use to spin up 10k emulated iPhones, to run a test in a CI job, for example?
At an old job of mine we had devices we controlled that could be rebooted. The initialization sequence after a reboot is much more network heavy than normal operation. We tried to push a critical security patch to all devices at once, meaning they all rebooted at once. Whoops self-DDOS.
It makes it very hard, sometimes impossible to upgrade a protocol, because some peer somewhere will try to interpret your new thing if it were a broken old thing; so security holes get enshrined. It is much better to fail fast, and make it easy to see why. The only people who benefit from enabling crap implementations, in the end, are makers of crap implementations. Their stuff should fail as soon as possible, so they have to fix it.
In some cases, it is acceptable have a "permit crap" mode that can be turned on in cases where the other end is not negotiable. But ideally the crap fails to work the first time it is plugged in, and returned to the manufacturer. Stuff that causes no end of trouble gets replaced eventually, but somebody has to expose the trouble, and it is best to have lots of company so the culprit is blamed.
maybe in the ivory tower of web shit, but it is essential for internet/society. like you said
> in cases where the other end is not negotiable.
aiming for robustness is key if you want to enable a diverse ecosystem.
We settled on throttling demand based on the incoming IP at the firewall...dropping incoming http packets for 3/4 of the addr space, then later 1/2, then 1/4, etc.
That was long ago, though, long enough that it was an NetScape webserver.
What is an efficient architecture for something like this?
Are millions of devices hitting some gateway->load balancer everytime the Workmanager wakes up and polling for whether new messages have arrived?
For example, if my phone does a "sync" 3 minutes from now, as do millions of others around the world, do we all get routed to our respective box that's keeping a queue of our pending messages in memory? If we happen to receive these messages in between the poll intervals via persistent socket or push notification, then the system wipes the in-memory queue.
Am I imagining this correctly or is there a more optimal architecture for something like this?
I woke up one Monday morning to crashed servers and angry phone calls once in that situation. Unfortunately it wasn't success on my part but what I had done the Friday before was so crap it couldn't handle a hundred users. But we blamed it loudly on unexpected success, quickly patched the mistake and everyone was happy. YMMV :)
Feature flag for maximum retry backoff time: https://github.com/signalapp/Signal-Android/commit/93e9dd642...
Add jitter to backoff time: https://github.com/signalapp/Signal-Android/commit/8f7fe5c3e...
Handle ServerRejectedException (HTTP 508): https://github.com/signalapp/Signal-Android/commit/c95f0fce6...
Feature flag for automatic session reset: https://github.com/signalapp/Signal-Android/commit/a3c7e7e55...
"A location for flags that can be set locally and remotely. These flags can guard features that are not yet ready to be activated."
Here is the file with javadoc on top:
https://github.com/signalapp/Signal-Android/blob/2c1c6fab356...
Solution: TRY-CATCH
Now after the fact of its popularity from the source of it all [1].
My response: Use Erlang.
Downvoters: So for a chat app that is still down for several hours and needs to be reliable to handle the scale of new users, what should they (Signal) have used instead? Java?
[0] https://mobile.twitter.com/ejpcmac/status/977192665489035265
[1] https://mobile.twitter.com/anildigital/status/13502749007375...
Secondly, the point of Erlang is that you do not need that many physical/virtual servers to scale to these millions of users. Perhaps just a few or the same number of servers they have, but running Erlang instead. The BEAM VM and languages based on this is quite frankly designed for this scale.
Still working well for WhatsApp and Discord.
[0] http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
The grandparent comment is nothing more than a cargo cult. "If we use the same language as WhatsApp, we'll magically be just as scalable as WhatsApp". It's horrendously naive.
Not sure what this section is about. IP connections are keyed on a 5-tuple of
{PeerAIP, PeerBIP, Protocol, PeerAPort, PeerBPort}
If you hold PeerAIP and PeerAPort and protocol fixed (single server IP, single port, TCP), you can accept 65535 connections from any other IP (so long as the system managing that IP can use all the ports; if it's CGNAT, it probably can; if it's a traditional OS, it will probably have soft limits, but easily thousands).
If your server is 192.0.2.23, and it listens on port 5222.
You can accept 65535 connections from 198.51.100.1.
198.51.100.1:X to 192.0.2.23:5222 for X in {1:65535} (port zero is weird)
You can also accept 65535 connections from 198.51.100.2
198.51.100.2:X to 192.0.2.23:5222 for X in {1:65535}
Pick any X, like say 12345; a connection between 198.51.100.1:12345 and 192.0.2.23:5222 does not preclude a connection between 198.51.100.2:12345 and 192.0.2.23:5222.
Thus, this only gets to be an issue if you have a large number of potential clients behind a single (public) IP address; which is generally not the case (except when you're testing from a load generator)
In fact it is the other way round and especially for Signal’s use case.
Signal is already seeking for Erlang expertise in their server developer roles so I’m sure they’re already looking into this.
Perhaps using languages that are not suitable or designed for distributed systems like Java sounds more like a cargo-cult than using Erlang.
They bought into the JVM hype until their server capsized on high traffic (again) and it is still down.
If you don't believe me, I dare you to create an alternative. I'd easily take bets that Signal's current system could easily outperform your cargo cult alternative written in erlang.
Maybe you should also apply that ’experience’ again by applying to their jobs posting here [0] as it seems you easily fit their requirements unless you doubt your own claims.
> If you don't believe me, I dare you to create an alternative. I'd easily take bets that Signal's current system could easily outperform your cargo cult alternative written in erlang.
So you’re expecting someone here to create an open-source alternative to Signal using Erlang for free? With that $60M in funding, Signal can easily hire someone who has ’first hand experience’ just like you to rearchitect their systems to be more reliable and fault tolerant.
The job [0] is still there. I can also bet you won’t help them out, even with your ’experience’.
[0] https://jobs.lever.co/signal/2a5fee8b-5875-46d4-a41d-773a28a...
No worries, It's understandable to not be in the mood to take a role where 10 to 20 million+ concurrent users already scares you to not bother applying anyway, despite your 'experience'.
> But even if I was, I'm not the one running around telling extremely experienced devs that they should rewrite their apps in a different language because of a "web scale" cargo cult.
So even when Signal is actually seeking those who have 'Deep knowledge of Erlang experience' in their server-side roles, In your view, they and I are 'cargo-culting'? Come on now.
> I have nothing to prove to them, unlike you.
Says the person who just yelled 'I have first hand experience' and then immediately bails out when challenged to apply said experience after 'daring' someone to create another competing open source messaging app using Erlang for free. Ha.
Sounds like it would have just been better to not even mention you have said 'experience' after all.
Granted its a lot more difficult to have 2m open connections in python. It is possible though.
You're also assuming that Signal uses the same connection scheme that whatsapp does, which I highly doubt.
Erlang would have been solution if they have predicted the brain-dead actions of Whatsapp/Facebook, and if they had ample time to prepare - and it would still have some hard limits on a number of messages it can handle.
Exactly my point. It’s still down. Their Java servers doesn’t seem to handle this scale well and aren’t reliable for this.
The fact that they have to “Add more servers” to handle this traffic tells you exactly why Erlang / Elixir or BEAM VM languages are better suited for this (And also can save them money instead of provisioning more servers on AWS).
> obviously they need a solution that works with the stack they currently have
It was down before even when they had less users. One user made this suggestion 3 years ago, and Signal looked at this. Plenty of time to switch.
> proposing Erland at this time is asinine.
You mean 3 years ago with their $60M in funding? Plenty of time and money to hire an Erlang / Elixir consultant.
You do realize that the major reason BeamVM is popular for this use case is because it makes it easy to add more servers, right? Erlang doesn't magically make it so you never have to add more servers. Actually, it's the opposite. Erlang is pretty computationally inefficient compared to most statically typed languages like Java. If there is any VM out there preventing your from needing more servers, it's the JVM. Generally speaking you can do more work with fewer resources with the JVM than you can with the BeamVM...it's just that the BeamVM makes it easier to add more resources.
Proposed fix: Rebuild the entire backend in a different language.
estimated time to outage mitigation: 6-12 months.
Now colour me "inexperienced" but I'd personally push a fix for exponential backoff to the client.