Incident with Issues and Pull Requests
githubstatus.com
githubstatus.com
It's a feature, not a bug!
https://www.wsj.com/articles/amazon-scooped-up-data-from-its...
The gist is that yes it’s true. They’d come out with their own Amazon Basics branded stuff and push it to the top.
I suspect that it's related to high load.
To be clear, I _was_ mapping previous incidents with this year's incident — no competitor or hard feelings involved. I really like Vitess, fwiw. And the only thing I really love is FoundationDB :)
Side note: "Autosharding" is largely a myth that unproven databases are touting. Sharding is complex and requires planning and control. Databases that start shuffling data round without oversight produce nasty surprises. Trying to be too magic is normally always a mistake with databases.
It was the DB, and it was rogue usage on May 10, so I'm standing by my original comment
I have an important fix that need to be deployed right now but there is no way to deploy it in a normal way with our CI which one was setup with Github Action. Fortunately I have a instruction to bypass CI and build the source by myself.
But again, Github defeat me because our release workflows are depend on GitOps which are effected by Github issue. Ahhhhhhhhhh I have to build the docker image, push it to ECR then update a YAML template to make EKS apply the new changes
It's 9PM in my timezone and I'm waiting for my patches are up. A frustrating incident
I'm interested in building the same but if yours already works well then I see no point in duplicating important work.
Here's a gist to the script: https://gist.github.com/sklarsa/845152721ee9292eb01f70756b89...
As for gitea, I'm just hosting it using Docker and orchestrating using a simple docker-compose file that maps the gitea data directory to a Volume on the synology: https://gist.github.com/sklarsa/0dd6d6094dac6bf6e7bf61df9ca5...
It's all hosted on my private network at home.
https://www.cnet.com/culture/windows-may-crash-after-49-7-da...
Imagine not only not being able to push your code, but also not even being able to _write your code_ at all. And so many orgs rely on Actions to even be able to deploy. Geez. I personally believe that the cloud sucks.
It's sill weird to me how many and how much companies relly on Github infra.
This is the error I was seeing:
ERROR:
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.Of course, if this continues happening it may be time for the OSS community seriously consider migrating en masse.
[Edit to fix grammar - thanks for the corrections!]
As you note, en masse is common — enough so that you even don't need to italicize it (although you can).
Really at this point, you just might as well consider self hosting and it is looking very chronic with GitHub falling apart and self-hosting was indeed the sensible idea just like how the other open source projects have done for years.
GitHub is going just great, and centralizing everything to GitHub really was a good idea wasn't it? [1] /s
(/s of course)
Right now I can ignore that my PRs show 500 error, or old code best case scenario.
But... I cannot build and ship due to project's dependency depending on some stuff hosted on GH.
Also hugs to any Devs, Opps or SREs directly effected by this outside GitHub.
Looking forward to a post-mortem on the last few days, I'm sure it will be a really interesting read.
The git+nginx would suffice but it does not offer GUI. I need one to see the changes proposed (aka PRs).
Gitea is nice, but a bit overkill for my needs. I don't need CI, files hosting, issues, team members, releases, wiki, forking/watching/staring, etc.
I'm just looking for "website" (read: interface) that list files over HTTPS, with the ability to show nice looking diffs. Some sort of ssh keys(?) to prevent unauthorized access etc.
https://github.com/sitaramc/gitolite
I've not used it personally though, so no idea how well it works in practise.
I just found about "git instaweb" (developed by Git authors, but it's extra OS package) - it works locally, probably what I was looking for (minus the 100% width page layout).
BTW There's Gitblit is someone's fancy hosting Java app.
I have not used it before, but it seems like it follows your requirements (Except auth using SSH keys maybe, but that could be a task for e.g. Nginx)
Depending on your needs, this can be as simple as sticking repos on any server you have and cloning/pulling/pushing over ssh. If you want something more sophisticated, though, there's a handful of nice applications (gitea is being suggested further up-thread).
That said, they're getting to the point where they really need to make some larger post about this. It seems reasonable to assume it is all from one root cause.
But strange that it keeps happening almost every day now.
In the past I worked at a company which used the commercial solution from JFrog, I don't remember ever having problems with git availability as a user.
There are aspects of permissioning that the cloud git repo providers have that become more challenging to implement as a home grown solution and unless you have the resources to maintain it, it also becomes interesting.
On one hand, you can do `git clone --mirror` and you'll have a copy of the repo and put that on a file share... though there's no permissions or automatic syncs for it (or CI). If you want those, then you get into some development (and maintenance) of the git hooks.
Going to things like a local hosted gitlab instance means that you need to have a local docker hosted environment running, and someone to maintain that, and the storage for it, and all of the other fun that comes with administering a complex 3rd party application on prem. When things are going good, it's an hour or two a week... when something breaks its several hours with calls to support (you're using a paid / licensed version to get support... right?) from someone who has a sysadmin skill set rather than a developer skillset. And don't forget about DR.
I'm not joking: https://github.blog/tag/github-availability-report/
It would show a very prominent zero and be a static page with no logic whatsoever.
Edit: 10 minutes later, the Github finally shows the push, but triggers still aren't working.
Edit #2: Things are working normally now.