Incident with GitHub Actions, API Requests, Git Operations, Issues and more
githubstatus.com
githubstatus.com
I hate that companies are saying degraded performance for "its not working at all" (even if only for some people). That's not degraded performance, that's broken.
At least the status grid says Partial Outage, but stop with the degraded performance bullshit.
Where this sometimes turns into a "partial outage", is that there might be response timeouts somewhere in the backend of the stack, meaning that things that took 60s are now taking [100s and then a 504 error response]. ("Because," the backend dev thinks, "why would clients want a response after 100s? They've probably given up and gone home already at that point.")
Or, equally likely, there's a response deadline somewhere in your client—where you figured "their API will never take more than 100s to respond", so even though the API is still chugging away at the 100s mark, your client cuts it off, drops the connection, and displays an error.
Either way, the literal problem at hand is still, simply, "things are going slower."
"Degraded performance" can help you predict whether you'll be one of the customers encountering failures, in a way that "partial outage" cannot. Were your particular querying workloads to the API fast before? You'll probably be okay. Were they slow before? Then you might experience the further slowdown as an outage.
If you want to design for fault-tolerance of this case, it's important to know how APIs you talk to deal with long-running workloads. Do they have timeouts? Are they customizable? Are they different per plan level? These are things the API's documentation should communicate.
Once you know these things—and you have metrics on how long requests to a given upstream API are taking to complete—you'll know exactly whether "degraded performance" will affect you or not, before actually getting customer reports.
This is a distributed-systems design faux-pas. Systems that need information to start up, should have some sort of burned in or persistently-cached stale version of that needed information, such that they can start up without being able reach out for it. They should then probe for the newest info until they have it. (See e.g. Elixir's tzdata library.)
If you don't do this, and instead put a synchronous remote call in the critical path for a service's startup, then you're turning "degraded performance" into a real outage condition, entirely unnecessarily.
It's actually not that common, because you need to get to a certain scale before things like this start happening, and that scale brings with it hiring people who know about scenarios like this. But sadly, it does still happen sometimes.
That's no longer a performance issue then though. It may have been triggered by a performance issue, but for downstream services or users, its a plain old outtage.
Besides, who cares if its "technically" a performance issue if its unusable for you? Its the end result that's important and if I can't use the service, it make no difference to me what the technicalities are, only that I cannot use it, and my customers certainly don't care if my product stops working because microsoft or amazon or whoever have performance issues, they only care that my product stops working.
Let me try again:
The reason sites report "performance degradation" in these cases, is that what can usually be detected immediately and automatically by metrics is the underlying performance degradation; while the fact that some process got unexpectedly wedged due to the performance degradation usually goes undetected for quite some time, until customers start reporting the outage. And, before the dashboard can be switched to "suspected outage", it has to first be determined whether any of these customers' reports are of a real "outage", and not just of "my own client is programmed to give up on the request after 100s" problems.
Each of these weird edge-case distributed-systems failures is usually fixed — stopped from ever happening again — after the first time it gets triggered. Which means that every time one of these happens, it's the first time the system has failed in that particular way. So there's never any automated monitoring in place to notice that the system has failed in that particular way — as that would imply that the failure was in some way predictable.
As such, while yes, the outage is real, I don't blame a company for not reporting the outage as such immediately, or for quite a while after it happens. Sometimes it's not clear right up until the root cause is resolved, that it actually was a partial customer-facing outage. Sometimes all your third-party downtime detector apps just aren't doing the right things to hit the code path / get routed through the middlebox that's failing for some customers; and all the real-customer reports getting through to you are vague and muddled.
(One thing that can be immensely helpful to a SaaS's ops team, is to have at least one highly-technical company as a customer who reliably only reports "upstream fault" problems, rather than PEBKAC problems. If that canary customer is complaining, then the outage is likely real. They're better than any downtime detector service — and, surprisingly, often faster, too! Sadly, most SaaS businesses have literally no single such customer.)
A waitress can't keep telling a customer their food is coming when they've actually lost the order ticket and have no clue what was ordered anymore, just because other customers aren't having the same issue.
Well, I guess she can in this case, and other customers will happily come to her defense!
A teammate once made a "problem" light for Gmail as a joke and coded it to be lit when there were any alerts firing at any severity in the internal monitoring system. I think most people would say that Gmail is pretty reliable, but if the bulb ever went out, you could safely assume there was a problem with the bulb or the monitoring system.
That's why big systems have SLOs like (to give a simple example) 99.9% of requests succeeding rather than 100%, or 99.999% of customer data being currently available rather than 100%. (Sometimes they can get more complex like at least x% of customers have no more than y% of time periods in which more than z% of requests fail.) And if they have per-customer SLOs, those are generally much less strict than site-wide SLOs and require a certain traffic level to take effect.
In today's case, I think github's problems are widespread enough that whatever SLO they've set is being violated, but the point stands in general.
It wasn't a performance issue though. It was literally just plain not working. This is the error I got when trying to push:
! [remote rejected] master -> master (failure)
Or worse: they might have given up on this request and issued another one. When you have a problem that might be caused by (or exacerbated by) overload and might cause heavy retrying (by the user and/or automatically at one or more layers), you want to give up on the requests that are least likely to matter. Deadlines are a standard way to do that. Other ways include propagating cancellation through the stack, sheddable request categories, and queue disciplines like adaptive LIFO.
Anyway, I agree with everything you wrote, and I want to point out that whatever deadlines they have in the system are probably helping more than they're hurting. Of course I can't be sure of that without knowing the nature of the outage.
But this goes beyond just this particular case. Remember when half the internet was down because of an AWS issue? Yet they still said "degraded performance" or similar.
Is it that what we are doing is becoming more complex, or are we, as a web, moving faster and breaking more things? I wonder sometimes if the advancement and affordability of our hardware and the systems we can run on them has decreased our reliance on optimization and correctness as a requirement of deploying on affordable hardware. Basic functions used to have to be combined with great rigor to run on the same machine for swaths of users. Now we are so far from that we have 1000 layers of abstraction, and it isn't hard to imagine how lots of tiny inefficiencies stacked on top of each other may be adding up to bigger issues.
In my org we have a term for this, where you're encumbered by faults deep in the toolchain - getting 'middlewared'.
My local taco shop ran out of rice a few weeks ago. A few days ago they had some unspecified kitchen issue and had to close. They don't open on sundays.
Occasional downtime is normal. We can try to prevent it, as devs, working with machines, but it is inevitable. Nothing continues working forever. All industries have downtime, it needs to be accepted and planned for.
Even in the old days, with the simplest possible thing that needs a db connection, you could outscale that db, or bandwidth, or whatever, and crash. It was easier back then in a lot of ways - 1000 people visit your new site? Crash. Before auto scaling infra you had to either deal with it or pay for too much hardware. I'm sure there are many other examples.
And those that can't - e.g. those with lives at risk over failure - have very slow, conservative and burdensome processes for changes and approval, that would never be accepted without the necessary risk.
Critical industries plan for failure. Pilots and astronauts plan for catastrophic failure (and it happens). Hospitals sometimes have fires or run out of oxygen or beds or have equipment failures and have to plan for that.
Everybody assumes tech will just keep working in the SaaS world. At the end of the day though, costs of the average service having downtime (even if you calculate google makes 2 million per minute) are generally pretty low and are usually recouped after the service comes back up.
And mine has never run out of rice. They are closed on Sunday and Monday. I don’t monitor them enough to know how many 9’s, but I’ve probably eaten there 200 times in 9 years and never experienced an outage.
Some places suck more than others. But my expectation is 100% uptime when it comes to rice at my taco shop.
I think assuming competence is healthy, as long as I’m not a jerk when it happens to fail. I love my taco shop and would survive if they didn’t have an ingredient once. But they work really hard to keep everything working.
The analogy falls apart when you say 'maintenance'. Maintenance tends to have fallback detour paths, is expected, and isnt a 'deal breaker' like this kind of outage is.
This is closer to an 'accident'. And if accidents are frequent at a road location, it _is_ considered an indictment of the civil engineering of that location.
https://www.githubstatus.com/history
There have been multiple incidents longer than an hour just in the last few months this year.
It's worse in some ways than, say, 5 years ago, because... GH offers more services. And more people end up relying on those, so, there's more than can go wrong. And when it does, it usually has a bigger impact.
Some client projects have tied everything to GH/Actions/CI/deploy/etc. When there's any issue with GH, something is affected. Other clients have more distributed services (bitbucket/gh/gitlab/etc), which is ... sometimes less convenient, but usually means something is still up and functioning even if something else is interrupted.
In a world of single servers - or even multiple servers load balanced over a single database - you only have a single point of failure. And while that is often touted as an terrible thing, once it goes down it's game over... in some ways, our scaled up distributed systems have a worse problem.
The issue is we envision our distributed systems optimistically - as isolated little bubbles that receive and emit clean, efficient, fault tolerant packets. When one goes down, only a few others get affected, and the broken module can be easily re-deployed with very little fuss.
In reality, the people with the vision quickly become horrified with what is produced. The authentication system touches everything. The stock management system gets hooked directly into the homepage. Suddenly, every single part of your system is hinged on at least 5 servers, and if any of them go down, it's game over until one of them gets brought back up, and by that point the job management queue is so backed up with critical tasks (that should probably have never been entrusted to a background jobs queue) that it takes hours for the organism to re-balance itself.
And so, these systems go down more often, because instead of one single point of failure, you have 5 - while telling yourself you have none.
The question is, how can we do this better? I've seen big teams and small teams similarly struggle with this. I guess Netflix with their chaos monkey strategy wins on this, but even that is not infallible - as you say, the 1000 layers of abstraction (and goodness knows how many unique combinations of interactions and potential simultaneous failures) something is bound to be missed from time to time.
So I don't think the problem is we're moving faster or breaking more things. It's just that we don't know how broken our systems are until they break.
Basically all GitHub services are currently degraded or down today.
Time for lunch. The push will get through eventually.
Doesn't matter, I wanted EMUs without the demo anyway. Still was funny to happen when I was talking to an engineer at the time.
Not really good if you don't have a backup method to avoid this. Unlike those who went all in.
> remote: fatal error in commit_refs
Seriously: for example copying then transforming any one of your local .git repo to a "bare" repository takes like two commands. Anyone with a Linux machine can do it. Then all the devs can temporarily use that repo to push/pull/whatever.
I think if you're to the point where you're emailing patches, it's worth it.
I'm confused as to the difference between https://github.com/nektos/act and https://github.com/actions/runner
Is it not possible to use 'runner' to run github actions locally? The docs for this seem like they are extremely sparse - probably because this competes with the paid option?
Much hugs to the GitHub SREs.
I guess I'll have to manually push fake commits to get it to update, or wait for whatever internal cache to drop off grumble grumble
I checked after none of my pushed commits showed in PR's, and it was "Webhooks" only. That's not useful.
This is different from commits, which probably should be Red.
Ran garbage clearance for nothing also. :-(