We are investigating reports of degraded performance
githubstatus.com
githubstatus.com
Or know which user to pull history for
> just the name of repo could be sensitive info
The phrase "sensitive info" tells me disclosing it to a mass code hoster bare any liability deems irresponsible. Remember Eric Schmidt about privacy on the internet?
Choose your priorities.
Why is “security” always a pissing constant with some people. I’d swear that I’d be condemned for locking my house at night instead of simply encasing myself in concrete for all eternity like Chernobyl. After all, it’s more secure!
Guess I can't treat github like a CDN.
Obviously I should have typed %
The current abomination I'm working on avoids this by caching the errors and serving them for several hours...
Even my open source projects in github are just mirrors from the "real place" of work: gitlab or my own gitea instance. If github is down, it is a minor inconvenience but I can still work.
Downvote me to grey-world if you like, but I think everyone's crazy to put all their code infrastructure in the hands of fucking Microsoft. Especially literal free open-source software. Who do you think Microsoft is? What do you know of Microsoft's history and their core values (they're "embrace, extinguish & exsanguinate"). It's like giving fucking Sauron safekeeping of your power-rings in Mordor, oh we have great infrastructure for safe ring storage here, very secure, the orcs are really expert guards.
ps. Sharepoint <shudder>
so it's actually very hard to switch
Only slightly kidding.
> The design of git makes switching to another primary remote very easy
It is never as easy as just switching to another primary remote.
It's not just Issues/Wikis/Releases, but the build/CI process(es) that are rampant on GitHub now, the community you've built potentially coming up on GitHub and not really getting that there's anything else out there, etc.
This is all time consuming hard work. People will often just not do it, and this is why we have lock in everywhere. Stop calling it easy.
The centralization of the internet around a few specific services - like GitHub, Cloudflare, and so on - is an increasingly problematic thing to contend with.
Edit: "as a easy" -> "as easy"
Last time I tried Act it had some pretty severe limitations. Perhaps I should take it for another spin.
Git hook management's really awkward. Even with tools to synchronize them (... all of them? You may want some that are just for your own use) it's a pain. "I want these hooks to run, in order, but only when a merge commit happens on a machine with such-and-such designation, and I want it to run the task on a different machine, but we need to make sure that runner's a Windows box because..." that just sucks to self-manage, and yeah, there's no standard for expressing that, you're bound to incompatible solutions.
Secret management's a hellscape and everyone's always glad when someone else solves the problem for you. That alone is like 50% of the value of Github Actions.
Are we talking GNU Make, nmake, BSD Make? (Isn't this why autotools exist in the first place -- to make something allegedly cross-platform for different flavors of Make)?
I get bitten repeatedly by sh being a symlink to different Shells, even though I've been using it for many years. The most recent piece of insanity being "cd foo bar" resulting in an error, but changing to foo directory while in some other version simply resulting in an error.
Also, error reporting. It's way too broken to consider sh a reliable tool. I wish things could be done with very simple tools and all this complexity around them was unnecessary. Unfortunately, here, this complexity while isn't unavoidable is indeed warranted due to abysmal quality of the simple tools.
These aren't very good at some of the things you actually want to use a CI for, other than the literal "build" step (which probably is using them anyway, or the per-language equivalent).
Coordinating _and visualising_ multiple builds where some parts run in parallel (e.g. multiplatform), or parts that might be different or skipped depending on how the flow was triggered. Managing caches and artifacts between stages and between runs. Managing running in different contexts, including generic shell and across multiple machine types. A consistent way to manage and inject secrets. Reusing common parts across different CI workflows.
I suppose you could have a parent job taking up a full slot to run a Makefile that launches and manages the running of jobs on other nodes, but imagine you'd have to step into some toolset that abstracts some of it, and hope that is shared development or you end up with an in-house convoluted nightmare of shell scripts.
"Something DAG and yaml shaped" is about the closest convergence we have gotten, and the closest that it looks like we'll get.
In CI, it now just needs to run the scripts in the correct order.
What's needed is a formally defined CI spec(s). Common is bad for the same reason any monopoly is bad. Formally-defined solves some of the same problems common is solving where it's important to be protected from random failures of the sole provider, but it also makes it, at least theoretically, easier to have multiple providers.
This is similar to how C is different from Make. C is a standard that anyone can implement, while make is a weird language defined by its implementation that some tried to reimplement, but by doing so only increased the insanity of compatibility issues.
Of course there were multiple attempts to make common / standard definitions for general-purpose automation tools. Make is one of those, Ant is another one, and there's plenty more. I'm not sure why none really sticks around to the point of becoming a universally accepted tool / standard. Some reasons I can think about are: languages build special-purpose automation tools around their implementation which are often the selling point of the language, an attempt to sell it to developers, so the authors are disincentivized from making them general-purpose. There isn't a consensus on what such tools should do and what kind of guarantees they need to offer. Some such guarantees may come with a price, sometimes very steep price, so would be hard to sell. Eg. something like tup offers highly reliable reproducible and isolated builds, but at a cost of complexity and resources, whereas something like Make offers no guarantees, but is easier to get started with and to be productive.
Maybe it could be possible to extract just the component of CI that deals with the "skeleton" of automation, defining abstract tasks, dependencies between them etc... but then immediately there'd be plenty of systems that'd try to complement this automation with their own (proprietary) extensions which would circle us back to the initial problem...
Otherwise any customer who gets a bill for more than a few dollars will replace them with a VM and copy of Jenkins.
So what I want is kind of to build a company that's just strongly opinionated about that... "everything is in one Git repository, the `main` branch is authoritative for ACLs and CI/CD config, the bugtracker writes issues directly into that repository and has an ACL role that allows it to do that, the RFC widget writes your code design docs directly in there as well, we do rebase but we merge without fast-forwarding, you have to use semver and it works like this..." and probably nobody will use our offering because "GitHub is more trustworthy" but "if anybody does they'll love us" haha
I've been meaning to give radicle a try.
Fossil lets you synchronize artifacts both ways, so you can clone a repo, participate in discussions when on a plane, write some replies, connect back to the internet and sync them back up to the server.
When your server goes down, you don't lose anything. There's no need to remember to back up your issues every day, every time you clone the repo, you also clone its tickets, forum, wiki etc.
It also pulls down anything i star into a different folder which get a sync one a day. The rest get a sync every hour.
Hobby projects and today's work? Sure. Point straight at GitHub and hack away. And when it goes down, get yourself a coffee.
But everything that's anywhere near production should have already pointed those github requests to a mirror or other tool in your own controlled ecosystem. The status of GitHub should have nothing to do with whether you're customers are getting what they're paying you for. Same goes for Docker containers and every other kind of remotely distributed dependency.
My cloud provider is probably an acceptable point. If every AWS region goes down I'm not going to have a spare cloud provider.
What about an auth provider? Do I need a backup there?
What about CI, do I need multiple CI systems?
3rd party search services, realtime messaging services, the list goes on.
For 1% of systems, you need backups for all of these (or to not use anything external). The other 99%, building backups for every one of these systems is a losing business strategy.
Some of them sure, but which those are will vary based on the context. It's not as simple as having a backup for "every other kind of remotely distributed dependency."
Everything you mentioned can be replaced. Your source code can't.
But that takes longer than the expected duration of this outage, and is a lot of work. It's not like we have a standby gitea we can just seamlessly switch to, so we are still hit by this outage. On the other hand for build dependencies we do have a standby mirror.
Relying on Google for that would probably be a mistake. With cancelled projects and cancelled users, that's high risk.
In the general case, adoping an external system will bring with it greater reliability than trying to run stuff oneself. The differences are that you don't get to choose your maintenance windows, and you can't do anything to fix it yourself.
Take care about who you pick, and own the depencency, because you've put a part of your own reputation in the hands of your provider.
Now, if you pick GitHub as a part of your controlled ecosystem -- which is totally reasonable, if it fits your use-case -- then you still shouldn't be pulling arbitrary stuff from places outwith your control. GitHub has package repository tooling that you can use :). Although it's not entirely clear to me that it's as suitable for third-party dependencies as tools like Artifactory or Nexus.
I've since read many sibling replies fret over build automation or source code storage, but some downtime in build automation will rarely damage customers and (my goodness) I hope people aren't trusting the only copy of all their IP to a business partner and that they always have a very recent backup somewhere they own themselves.
There's still a question of how strong an ownership we need -- for example, my mail server is owned by OVH, and its backups are sent to rsync.net. I consider that to be sufficent ownership of backups. I'm much more likely to lose my files than they are.
I'm sure that GitHub also have backups, but they're not for the benefit of their customers.
Seriously people: gitea exists. Gitlab self-hosted exists. Drone/Woodpecker CI exists. It's not that difficult to set up a project that does not depend on Github. I spent less time setting these up than the amount of down time that Github has had this year.
You don't ever escape the payment cost of the issues of keeping these systems running. You're paying staff to do it, you're paying a cloud provider to do this, or you're paying out of your own time.
You're engaging in survivorship bias. One of my first 'social' media/forum accounts accounts was lost when the admin of the site dropped the user table from a database, and had to go back to everyone online and ask for people to mail in about their accounts. I never bothered to set it back up.
Piles of other smaller sites disappeared for similar reasons. If you completely ignore all that, then yea the old internet was fine. Oh, and that these days hackers are highly motivated to encrypt all your crap for bitcoin.
Typically big sites don't disappear because they depend on separation of duties between different teams doing their responsibilities. For example, knowing how backups work and making sure they are working every day.
Today if I were using github, my day would be wasted again.
For all the talk about companies trying to cut out on meetings by putting a sticker price to it (this 30 minute meeting could've been an email and cost $2000), at what point do we start saying "this outage could've been avoided and cost us $5k"?
Times that by the the number of external systems you rely on and you start seeing why we outsource that stuff in the first place.
And I can take part of the "savings" and contribute to gitea's further development.
⮕ git push
ERROR: Repository not found.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Sighed, assumed I was on the first wave of an incident on top of their current slack incident, then logged off for the day.
People are going to jump on them no matter what. Having a engineers talking to PR wouldn’t help anyone.
My definition of degradation is different.
we couldn't onboard a new hire because they couldn't even run a curl command for some basic tooling
I thought it was a url, but the url actually loads a bash script that tries to pull from their github repo
Recently github is pushing very agressively for two factor authenticartion.
So I installed the authenticator app.
But the authenticator does not work when the clock on my phone is not perfectly synchronized. But my phone's clock is intentionally sped up by +15 minutes?
What to do?
Or you can save your code and wait 15 minutes to login.
There must be at least 7-8 other people in the world would pay for this.
Technically if an authenticator app has an option for a time offset it should work, but I've never encountered one.
Who decided to deploy something risky on a Friday /s
Why don't they have lower environments like QA to test it in before going to production /s