Can't they at least fix their status page? https://www.githubstatus.com/ It returns `All Systems Operational`. I mean what's the point of having a status page if it returns wrong info?
1. It clearly indicates that automatic systems are failing to detect the outage. 2. It also indicates that no one is aware about the incident to manually signal the outage (or that there is no manual override).
Basically, it makes a difference between "yeah, shit happened, we know (and maybe working on it)" and "hah, they don't even know themselves".
Partially because these large systems have some kind of ongoing issue at any given time, so it's challenging to provide a meaningful live status that isn't a bit misleading and could cause misdirected panic.
Partially because you don't want to give potential attackers (eg ddos) any insight if/how their methods are affecting your systems.
Partially because there are SLAs and reputation at risk, and you don't want to admit to any more downtime than you absolutely have to.
Of course, in the real world, I don't think there's a single IT admin who didn't just start nervously sweating and pulling at their collar after reading the above, imagining it as something their CEO said and encouraging them to target it. Nobody can really do this — and that in turn says something important about how those metrics really look in most systems.
To be fair, redditstatus.com is quite nice with their sparkline headline metrics. It at least lets you know _something_ is happening even if they haven't yet declared an incident.
They can do better, they just don't want to.
Besides that, how are you going to cause "more harm than good"?
A status page is a kind of PR. Think of it like a policy for a flight attendant to come out into the cabin to tell everyone what's going on when the plane encounters turbulence. That policy is Public Relations -driven. You only do it if you expect that it's positive PR, compared to not doing it — i.e. if telling people what's going on is boosting your reputation compared to saying nothing at all.
If a status page just makes your stakeholders think your service is crappy, such that you'd be better off with no status page at all... then why have a status page? It's not doing its job as a PR tool.
But that's not true. The company could do better. But the individual employees cannot. The individual employees are constrained by the profit motive of the company. They are not allowed by corporate policy to set up automatic status updates, for about the same reason they're not allowed to post their corporate log-in credentials: that the result would very likely be disastrous to the company's bottom line.
(Though, really, the corporations in most verticals are in a race-to-the-bottom in most respects. Even if you treat GitHub as a single entity capable of coherent desires, it probably doesn't desire to avoid automatic status updates. It needs to avoid them, to survive in a competitive market where everyone else is also avoiding them. People — and corporations — do lots of things they don't want to do, to survive.)
GitHub, meanwhile, is a million different things to a hundred million different people. Users of e.g. Homebrew, with its big monolithic ports system hosted as a github repo, have a very different SLI for Github than do users of some language-ecosystem package manager that allows you to pull deps directly from Github; than do people who depend on GitHub Actions to CI their builds on push; than do people doing code-review to others' PRs; than do people using Github mostly for its Wiki, or Issues, or downloading Releases, or Github Pages, or even just reading single-page-with-a-README repos, ala the various $FOO-awesome projects.
For many of these use-cases, Github isn't degraded right now. For others, it is.
If you ask for Github (or any service with this many different use-cases and stakeholders) to measure by the union of all these SLIs, then the service would literally never be not-degraded. In systems of sufficient scale, there's likely no point where every single component and feature and endpoint of the system is all working and robust and fast, all at once. Never has been, never will be.
And anything less than just going for the union of all those SLIs, is asking Github to exercise human judgement over which kinds of service degradation qualify as part of their own SLOs. Which is exactly what they're doing.
Certainly, internal to services like this, there are all sorts of alerting systems constantly going off to tell SREs what things need fixing. But not all of those things immediately, or even quickly, or even ever, translate to SLO violations. There are some outlier users whose use-cases just break the system's semantics, where those use-cases just aren't "in scope" for the SLO. As long as those users are only breaking the system for themselves, the degradation they experience won't ever translate to an SLO breakage.
E.g., a bit tongue in cheek:
> An MMO is a very simple system, in that there's only one Service Level Indicator (SLI) that devs, shareholders, and players all agree on. That SLI is "can a player connect to the server, and perform regular gameplay actions, without a ridiculous amount of per-action latency."
Wouldn't you say that in an MMO of sufficient scale there's likely no point where every single component and feature and endpoint of the system is all working and robust and fast, all at once?
> In systems of sufficient scale, there's likely no point where every single component and feature and endpoint of the system is all working and robust and fast, all at once. Never has been, never will be.
Couldn't we redefine SLIs as "can the user connect to the server and perform regular user actions without a ridiculous amount of per-action latency"?
My point was that Github has no "average user." Github is like Microsoft Word: each user only uses 10% of the features, but it's a different 10% for every user. Yes, there are some features that are in the critical path for all users (loading the toplevel repo view in the website might be one); but for any given particular user, there will be plenty of other features that are also in their critical path.
An MMO, meanwhile, does have an "average user"; in fact, MMOs have ideal users. An MMO's goal is to induce every user (player) to play the game a certain way, so that the company can concentrate their resources on making that particular play experience as polished as possible. There is, per se, an idiomatic "rut" in the road that players can "click into", ending up doing exactly the same short-term game loops that every other player before and after them has also done when playing the game.
MMOs can be reduced to a single SLO: can the ideal player have fun playing the game at the moment?
GitHub cannot be reduced to a single SLO, because there is no ideal user of GitHub. There are probably two or three thousand separate "ideal users" (= critical, non-universal user stories) for GitHub.
> Wouldn't you say that in an MMO of sufficient scale there's likely no point where every single component and feature and endpoint of the system is all working and robust and fast, all at once?
No, not really; MMOs have a complexity ceiling by operational necessity. They aren't composed of a ridiculous sprawling array of components. They might use Service-Oriented Architecture, but in the end, you don't scale an MMO vertically by making more and more complex clustered systems with master-to-master replication and so forth. You scale MMOs by either pure-vertical hardware scaling, or by horizontal shared-nothing sharding.
(The key thing to realize about MMO servers is that they're OLTP servers — they need to track a whole bunch of users doing a whole bunch of simple actions at once; and therefore they can't really be doing overly-much computation on those actions, lest they lose shared-realtime verisimilitude.)
Not sure if any MMO reaches Github level, most likely not. But I don’t think there is a ceiling or any sort of hard distinction; i.e. I think 5 years from now we could have a MMO with complexity of today’s Github. Maybe it will be called a metaverse though.
EVE is literally the only exception to "MMOs scale by horizontal shared-nothing sharding"; and that's why I mentioned the option EVE uses instead — namely, "vertical scaling of hardware" (i.e. having a really honking powerful single-master multi-read-replica DB cluster.)
In neither case is anything "clever" (i.e. inefficient for the sake of developer productivity / enterprise integration / etc.) happening. There's no CQRS message queues, no async batch writes, no third-party services halfway around the world being called into, no external regulatory systems doing per-action authorization, no separate "normalized data warehouse for OLAP, denormalized data for runtime" forking writes, no low-level cross-replicated integration between a central multitenant cloud system and individual enterprise-hosted tenant silos, etc etc.
> With so many different play styles and optional components (guilds, pvp, official forums, paid content, user made content, etc)
I think you misunderstood me when I said that there's a rut that users are guided into. The thing about MMOs is that the ideal user uses all the features (because the game incentivizes doing so, and because the more deeply and broadly users engage with the game's systems, the higher their retention / lower their churn will predictably be.) The ideal player is in both a party (or constantly switching parties) and a guild; has paid for all the DLC and regularly buys cash-shop items; plays every piece of PVE content you build; engages in PVP content and co-op UGC content all the time; etc.
Which is to say, for the ideal user, "the game" either works or it doesn't, because "the game" is the whole thing. Every feature needs to work, in order for the game to work. Because the ideal user engages with every feature. The SLO is, essentially, "can you do a completionist run through every bit of content we have." (If you're clever, and can make your server deterministic, you can create a completionist run as a backend-event demo file and run it in CI!)
And this is, in part, why MMOs are kept architecturally simple. Everything needs to work!
(And I don't just mean "simple" in terms of the backend not being a sprawling enterprise-y mess, but rather usually a single monolithic binary that can keep a lot of state in memory. I also mean "simple" in terms of as much of the game as possible being pushed to local, mostly-ephemeral-state client-side logic. MMOs are, often, a lot less "online" than one might think. It's very hard to "break" an MMO with a content update, because most content updates are to zonal scripts whose state doesn't persist past the lifetime of the in-memory load of that zone in a particular interacting device.)
With GitHub, their ideal users — of which there are many — can be individually satisfied by very small subsets of the system, such that they're still satisfying almost all their users even if one system is horribly breaking. That's what an SLO is "for", in the end: to tell you whether different subpopulations of users with different needs are happy or not. If you only have one "core" subpopulation, one ideal user, then you only need one SLO, to track that one ideal user's satisfaction. If you have more, you need more.
HN seems to be going down for the massive amount of requests!