Skype brief post-mortem
skype.com
skype.com
As other people have stated on other articles posted here, no discredit to them for having some downtime (I'm sure they're working round the clock on it), but I have to wonder if this could have been avoided with a phased roll out of new version to the super nodes.
And would still keep trying to serve the requests, right?
So if sufficient number of supernodes is brought back online then the problems should disappear.
I assume that overloaded server still serves some requests. But my assumption could be wrong, so if you have experience with that -- please share your knowledge.
Another thing that baffles me: if I introduce new supernode that is sitting on new IP address -- why would all the traffic suddenly hit that node?
Shouldn't it be just gradual increase in requests while more and more Skype clients discover that new supernode?
As to your second point, I think you are right. I assume that is what the mega-supernode is: a network of machines who's resources are as high as can be to handle all of the connections and try to beat the bottlenecks.
See http://en.wikipedia.org/wiki/Thundering_herd_problem
Many types of systems need a warm up period before they can realize their full performance. In web applications a controlled warm up is often needed to prime the caches. In P2P applications - which you can't easily "reboot" as a whole - the restoration of a steady state can be much more complex.
Shouldn't it be just gradual increase in requests while more and more Skype clients discover that new supernode?
In theory, yes. In practice this seems to be a case of http://en.wikipedia.org/wiki/Cascading_failure
The remaining supernodes either can't handle the aggregate load alone. Or they are being overwhelmed because the re-connection attempts from clients are not evenly distributed.
Shouldn't it be just gradual increase in requests while more and more Skype clients discover that new supernode?
In theory, yes. In practice there's probably a lot of http://en.wikipedia.org/wiki/Positive_feedback and perhaps even http://en.wikipedia.org/wiki/Monster_wave going on in the skype network right now.
The GP's question is valid: there's no reason why server software should crash when overloaded instead of simply degrading service.
Made-up Scenario: My cluster can handle XXXk users with an SLA of YYms response time. In degraded mode, I'm only handling XXk users with YYYYms response times.
I'm not meeting my SLA for the remaining number of users so I am, in essence, offline.
As to your specific point of "crashing", look at what happened with 37s. Should the server have crashed? No but there was a bug. The reason you add more capacity in the FIRST place is because the existing number of nodes cannot handle the volume. Depending on any number of bugs, issues or configuration your degraded capacity is for all intents and purposes "crashed".
Made-up scenario #2: A single server in your apache configuration can handle 200k concurrent connections reliably with fast response times. Double that load and response times are so long that various devices on the path are timing out the connections as stale. Apache hasn't crashed but it's not really doing anything.
Fast failure is an accepted best practice. Shit, it's baked into Erlang. Kill the process, start a new one and move on. Depending on the nature of the crash, you're doing nothing but churning processes and not actually servicing requests.
The bigger problem is that people don't design for this type of scenario. Static landing pages. Decoupled services instead of monolithic all-in-one containers. Look at github. That's an awesome example of how to degrade service during an outage. Only certain components are "crashed" because everything is fairly decoupled.
Meanwhile there's a guy over here running 4 apps in the same tomcat container that communicate over memory transport with each other or even if he had the common sense to decouple each app, didn't bother to fail fast and was busy spinning up threads trying to communicate with the rest of the services that he can't actually respond to anything externally.
Degrading service, or more accurately, serving the maximum number of clients you can serve and no more is absolutely not the same thing as being down.
Failing/crashing rather than limiting the number of clients you serve is absolutely not established best practice. It's poor practice, in fact.
The skype outage probably not. This was in response to the thread going on about overloaded webapps - thundering herd is a common issue there.
The GP's question is valid: there's no reason why server software should crash when overloaded instead of simply degrading service.
Nobody used the word "crash" but graceful degradation is quite hard. You have to understand the problems at heart, plan and build for it. That is more than enough reason for most apps to not degrade very gracefully.
Graceful degradation is not hard. Serve N clients, and no more. It's an extra conditional in the event loop. Am I serving N or more clients right now? Then don't accept new connections (or accept(2) them and immediately close them to avoid a backup in the listen queue).
The accept()-issue you're referring to is merely a popular manifestation of the thundering herd problem.
The term is nowadays also used in a broader sense to describe similar situations in distributed systems.
It's not a common reason why servers crash when overloaded.
Again, nobody is talking about servers crashing here.
We are talking about web applications (which usually consist of multiple servers talking to each other) degrading to the point of not serving a meaningful rate of requests anymore.
In that context the role of the "thundering herd" is often taken by your own users, who will furiously smash their reload-buttons to get past those 500-errors.
It's an extra conditional in the event loop. Am I serving N or more clients right now?
You're talking about implementation details of a socket server. I was talking about entire applications.
When servers are getting overloaded, you need to throttle the incoming requests aggressively and early, to keep response time reasonable.
If they all disappear, I doubt there's a contingency plan for that.
So they can inject new supernodes in to the system and clients will discover'em gradually. That kind of indicates that they have a mechanism not to DDOS the new mega-supernodes. It's a contingency plan. I fail to see what we are arguing here.
Skype doens't own the supernodes.... skype works because it uses the user's own resources to help route calls for other users.
The fix should probably be about developing deploying new version of Skype client.
Such fix can take a week. Especially considering that engineering might be not in a good shape in Skype few years after acquisition from original founders.
As in all systems, things will go wrong, whether it's your own bug or someone else's. Your system needs to not only be stable in the steady state, but to be able to return to the steady state when something interrupts it.
Protocol: we used a custom psuedo-TCP protocol built on UDP based on libjingle from Google (used in Google Talk): http://code.google.com/apis/talk/libjingle/file_share.html. Libjingle's filesharing itself was a decent resource for learning some P2P stuff as well.
NATing and routing: we used Skype's UDP hole punching: http://www.h-online.com/security/features/How-Skype-Co-get-r....
Skype configures each install to be a supernode by default and then disables it based on certain criteria (low speeds, behind a NAT/firewall, etc.). Discovery of other network nodes is an entire subject unto itself and a lot of other network discovery protocols are well documented; our system used a central server to track up-time statistically and map clients to one another based on finding a good fit. Happy to answer any questions here or at my e-mail address in profile.
"An Experimental Study of the Skype Peer-to-Peer VoIP System" http://saikat.guha.cc/pub/iptps06-skype/
So, what was the problem affecting versions of Skype?
Your quote was an "aha" moment for me.
What a brilliant solution.
Edit: I initially read your post as sarcastic, but I realize now it may not have been. It's definitely outside-the-box thinking and after considering their options, this certainly seems the most viable.
Why would supernode that has CPU as a bottleneck stop serving _any_ requests?
Say, CPU is able to serve a thousand requests per second. If supernode receives a million requests per second, then supernode should serve first thousand requests and then ignore the rest, right?
But I guess overloaded supernode does not serve anything at all. My guess would be that after 1000 successfully served handshakes clients send 1000 data requests, and most of these data requests are ignored, because supernode is overloaded. End result -- almost no data is distributed.
Most likely the problem is in Skype code for supernodes (almost all Skype clients are supernodes).
I think we should expect Skype hotfix soon.
As regards the aforementioned 1991 failure of the AT&T long-distance system, which resulted in a service outage of about six hours, the document says:
"The PSTN's SS7 control network provides an interesting example of what can go wrong with a tightly coupled complex system. Outages such as the well publicized 1991 outage of AT&T's SS7 demonstrates the phenomenon: the outage was caused by software bugs in the switches' crash recovery code. In this case, one switch crashed due to a hardware glitch. When this switch came back up, it (plus a reasonably probable timing event) caused its neighbors to crash When the neighboring switches came back up, they caused their neighbors to crash, and so on [NEUMANN] (the root cause turned out to be a misplaced 'break' statement; this is an excellent example of cross- layer coupling). This phenomenon is similar to the phase-locking of weakly coupled oscillators, in which random variations in sequence times plays an important role in system stability [THOMPSON]."
Synchronized bug in that software – some sort of clock overflow, or update from Skype gone awry?
Or a flaw in that software discovered and exploited by others?
There must be a lot more to this story.
I have a paid subscription for international calls and my voicemail, my dear friend. Do you? If not, then stop mocking people you don't know. If you do, preferably also stop mocking people you don't know.
Having said that, I have important business calls to do today and I'm certainly not going to throw money at my iPhone carrier. I hope Skype is coming back online for me, soon. But I have not yet started complaining anywhere, because Skype has offered me a great service and lots of savings in the last two years.
But who am I to day, I'm just a grumpy German.
The only guess I have that isn't from an obvious reason is that Skype HQ is in Luxembourg, a small country right next to Germany. One of the three official languages there is German, and the other is Luxembourgish, which is closely related to German.
A principios de hoy, nos dimos cuenta de que el número de usarios conectado a Skype empezó caer. Esto no es normal ni esperado así que nos pusimos a investigarlo.
Skype no es como un red de telefonía convencional ni red de mensajería instantánea - en cambio, para que todo vaya bien depende de millones de conexiones entre los computadores y los teléfonos. Unos de esas computadores se llaman "supernodes" - funcionan como una agenda de direcciones por Skype. Si quieres hablar con alguien y Skype no lo puede localizar inmediatamente, tu computador o tu móvil intentará localizar un "supernode" para descubrir la manera de contactarle.
Normalmente hay un montón de supernodes disponibles. Hoy, por desgracia, muchos no eran conectados debido a una problema que afectaba algunos versiones de Skype.
¿Qué estamos haciendo para mejorar la situación? Nuestros ingenieros están haciendo nuevos "mega-supernodes" lo cuanto antes, que con un poco de suerte va a hacer que todo se normalice.
The explanation doesn't ring true to me because AFAIK supernodes are used only if a direct P2P path cannot be established between caller and callee.
What we're observing is the inability to sign in and see contacts' status. I always thought that was centralized. Are supernodes involved in signin?
Generally it's either a skype or ISP owned machine
http://www.h-online.com/security/features/How-Skype-Co-get-r...
As I understand it, for corporate firewalls or routers/firewalls configured to not allow "hole punching", Skype will use the supernode as a relay for all communications.
So there's something else they are hiding. My Mac skype dies on start since yesterday (trying to sign-in), supernodes would have nothing to do with that.