GitHub incident – now resolved | Hacker News Reader

GitHub incident – now resolved

githubstatus.com

73 pointsxvello75 comments

When I decided to move away from self-hosted git ages ago and then from Jenkins to GA two years ago, reliability was a huge factor in my decision because Github, I supposed, would be much better at keeping their infrastructure running than I am at keeping ours.

Turns out the uptime of both our git server and even the Jenkins instance beat GitHub by far and while the former only cost a marginal amount of CPU time on infrastructure I was running anyways, GitHub is a noticeable expense for us.

Of course it still saves me from the panic attacks every time I'm compelled to press the "Update now" button in Jenkins because either I do nothing and get my instance RCEd or I do press the button and who knows what plugin update will break which part of our setup, but while that was a constant fear in my mind, the amount of downtime caused by Jenkins plugin updates was zero whereas what GitHub is doing lately is way, way, way worse than zero.

I'm starting to get frustrated and like I presume many other paying users, I think I'm at a point where I feel like we should get partial refunds of our subscription money given the very spotty uptime all year now.

PSA:

Never expose Jenkins to the public internet, make sure it's via VPN. If you need webhooks, there are services for that which allow you to broker webhooks whilst calling in from the Jenkins side (i.e. not exposing ports). Even so if you do have to use native webhooks, at least lock it down to the upstream's IP range(s).

Ideally have a dev jenkins to test all the things first before hitting upgrade on your prod instance and killing some plugins (hell even better if it's all IaaC and can just spin up a jenkins host per env, but ££$$$££/Time etc)

Nowadays, tailscale or cloudflare access + tunnel works amazing well for private service that you might need access on untrusted network. So the needs for keeping them up to can be delayed a lot more (of course, jenkins is a special case since it might be pulling and executing untrusted code, but I think that is something you need to care even without security issue specific to jenkins itself).

Yea, quite right, also use runners on ephemeral build targets as that can reduce some of the attack surface for running untrusted code, don't do it all in the box.

We have open tickets about the SLA being broken for Q1 and Q2. They've been open for a while, and we're out of ways to escalate them (despite enterprise).

And GitHub's SLA is not great to begin with: 10% of spend refund at 3 nines (99,9%) and 25% of spend at 2 nines (99%).

In general, don't use Jenkins. It is mediocre piece of software with an even worse ecosystem. It self-implodes every now and then. Big tech companies build a lot of custom stuff to get it to work correctly. Once that is done sure, it works. I am not a fan of Github actions but to host GIT and Jenkins is not really a serious option. Pick another CI platform in that case.

This comment makes a lot of assertions without any backing data.

How is it mediocre? Is it because of the CVEs that have been released in the prior years? I recall GitLab also having quite a bad week of CVEs in February[1].

How is it a bad ecosystem? If this is about plugins in order to do things, I actually like this framework - it lets there be specific owners for portions of the open source development.

Self-implodes? This seems like it would be tracked as a bug. I've encountered an instance where Jenkins wouldn't start due to a crypto issue but that was due to a bug and all I needed to do was install a patch.

I think that using Jenkins can be a thought of a serious option if like anything else, you follow security protocols ie: don't allow public access, maintain RBAC standards, have a maintenance schedule.

[1]https://about.gitlab.com/releases/2022/02/25/critical-securi...

I have stolen access to production systems though Jenkins. Cause the scrambled pw is always sent to the Jenkins client. And then by default anybody could decode it from the console… They fixed it in 2.0 I believe.

The problem with dependencies is that different plugins need different versions but there is no solution to that. Even more. You update Jenkins. Plugins break. Data gets corrupted. You try to align the plugins to some dependent version. It fails, cause most of the ecosystem is a hobby-project that never gets updated.

Jenkins has absolutely nothing to do on the public Internet as most of these tools.

There are so many CI/CD services. Most are very cheap. You really need to love yak shaving to pick Jenkins over something from the shelf. No major cloud offers Jenkins Saas. Wonder why…

This is an odd response in my perspective.

'I don't like it because it's not secure and I was able to get access to prod via this method.'

'That method has been fixed.'

'Well...I still don't like it.'

Bugs are being found and fixed all the time. As for a the different plugins need different versions. Yes. This also occurs in general coding to where every package or library that you want to use in your code is also going to be versioned. If that package or library gets updated, your code might break.

This goes back to having some things in place. Backups and a maintenance and testing schedule. If you don't test things before pushing them out to prod maybe fix that first.

I have always been happy using GitLabs CI/CD tooling [1]. Also, the integration with the source code this way is like Github with Github Actions.

[1] https://docs.gitlab.com/ee/ci/yaml/gitlab_ci_yaml.html

GitHub is running a complex, planet-scale product. I think w they've crossed the threshold where doing it yourself is more likely to be reliable for some use cases.

We've been running GitLab on GKE for the past three years, no problems outside of initial migration pains.

Also: "you grant to Microsoft a worldwide and royalty-free intellectual property license to use Your Content, for example, to make copies of, retain, transmit, reformat, display, and distribute via communication tools Your Content on the Services."

https://www.microsoft.com/en-ca/servicesagreement/upcoming.a...

Am I missing something, or is GitHub distinctly not listed in the Covered Services section of that services agreement?

Nope, looks like I missed that. My mistake.

So what part of the github terms gives them permission to sell my IP back to me through copilot?

https://docs.github.com/en/site-policy/github-terms/github-t...

GitHub has a separate policy...

Thank you, I missed that part.

As I asked elsewhere, where is the part of these agreements that permits them to profit off my IP through copilot?

If your code was in a private repo they supposedly didn't use it for training. I think it was also mentioned they tried to limit it to only repos where they could determine some kind of license they thought would permit such usage.

If you're publishing your IP to GitHub in a public repo and using a widely permissive license, they didn't need an agreement to use that IP to train copilot.

There needs to be a line somewhere that says, "if you use a widely permissive license, we will use your IP to make money for ourselves" or something like that, then, right?

That's what I want. To find that line.

Do they need to have such a statement? What makes them need to make such a statement? You've given them the permission with the copyright license you've published your code with.

Yes it's required, at no point did any license I assign my code give them permission to sell my IP as a part of copilot.

Which licenses do you use and where in the licenses does it state it couldn't be used as training data?

My license says something along the lines of "This is totally free, except for use by Microsoft and Github in anything they develop or sell."

So, where does it say they can use my IP?

Cool. Call an IP lawyer, sue Microsoft, and enjoy retirement on your new boat.

Or I guess you didn't really have any IP of value that was infringed on whatever license you claim to have made.

Dang, I was hoping you or someone could help. Anyways thanks for participating in the discussion none the less.

If you're not just being facetious then that someone who can help is an IP lawyer, and I congratulate you on your new boat. I'm not being fake with that.

Buf if you published your code with an MIT or GPL kind of license chances are they don't owe you anything nor do they have to say that it's using your code. IANAL, but nothing in those common licenses leads me to believe they owe you even that statement. If I'm wrong feel free to point to it.

You're not really engaging in this discussion much more than giving extremely vague hypotheticals and then making demands that seem like quite a leap. Where in, say the GPL, would they have to make a statement like you claim they need to? What are the actual terms in your public license and code that you have that they would have infringed in some manner? The repo would have to be public and you just said it would be free, so please do link it. Otherwise, what are we even talking about here? Just making noise on the internet I guess?

That's now the second time you've disparaged me. I think you're wrong and now I also think you're a jerk.

Its disparaging to ask for the public codebase that you claim Microsoft is using inappropriately or for you to explain why you think they owe you something?

And how is it disparaging to suggest that if you really think Microsoft somehow harmed you that you go and do the thing that would actually resolve the issue?

I try and learn from my mistakes. Can you describe exactly how I am being disparaging by these comments?

Also, following up, will you share the codebase in question? You said its free, and its clearly something publicly hosted in GitHub otherwise its entirely out of scope from this conversation.

> When I decided to move away from self-hosted git ages ago and then from Jenkins to GA two years ago, reliability was a huge factor in my decision because Github, I supposed, would be much better at keeping their infrastructure running than I am at keeping ours.

Oh you sweet summer child

What's annoying about this is that the PR doesn't even say it's trying to run tests. It says everything is passing and just doesn't list the actions.

For a second I thought someone must have deleted the actions yaml files.

This is a dangerous failure mode.

https://github.com/multiprocessio/dsq/pull/82

Screenshot here: https://twitter.com/phil_eaton/status/1542168020516216832

As in you have it configured to prevent merge until they passed, they're not running at all, and it's allowing merges?

That's right. This button shouldn't be green. But it's not just that the actions aren't running but their service that reports that actions exist must be down too. That is a bad design. They should still report that the actions exist even if they can't run them. This PR button shouldn't be green/pressable.

There's a way around this: Under "branch protection rules" in the settings for the repo, you can require certain "status checks" to be green before merging. This has as an added benefit that it works for allowing "auto-merge" (at the settings page of the repo).

(I work at Microsoft, but not at Github)

Yes that is set already and that's why this is so concerning.

You can set Actions to required and then they will be displayed even when Actions is down and merge will be prevented.

It seems a little early to call it "now resolved." I'm still seeing issues. If it's gotta munch through a queue or something, it would be helpful to announce that info.

Edit: It's just the HN title that says "now resolved." This github status says:

> We have identified the source of disruptions and are actively working on a mitigation. The systems are in recovery and services are returning to green.

Title says "now resolved" but the link says the incident is ongoing and they "are actively working on a mitigation."

Agreed, the admins re-titled it too early. Git operations and API requests are back to green, but other subsystems are still impacted.

I have been so happy to be on GitLab again after some time working at a company that used GitHub. The issues and epics in particular are so much better, and CI seems to be more reliable.

So, it seems like there is at least once a week a partial outage on github. For how long ate the CTOs and Engineering managers all around continue to accept this?

Who shouldn't at the very least donate a bit to the various open source CI solutions, as a way to have some kind of hedge?

Highly recommend https://builds.sr.ht. I've been using for ~2 years, never had an issue.

Just recently I found out about sourcehut. The project is refreshing. I don't know if there is something similar in the market but I enjoy how simple and straightforward to use it is.

LMAO, the entire front page of SourceHut is broken.. that's not a good look...

What is broken? Both https://sourcehut.org/ and https://builds.sr.ht/ renders just fine to me.

Looks like it just fixed itself. I wasn't able to see any featured projects & search for any projects for a good 5-10 minutes there.

How is it broken? sr.ht? It seems to work for me...

Probably not the best use of archive resources, but you could store a snapshot of it, so hn commenters can believe you.

I really want to like GitHub Actions. But it feels like every time I'm trying to get something done, they are broken.

The problem that I've found is that no other CI/CD provider has feature parity to GitHub Actions & integration with Git. Sure, there's external providers (Travis-CI, Buildkite, etc) but none of them feel like they have the polish of GitHub Actions. GitLab & Azure DevOps also don't compare at all - I've migrated whole organizations off of both because they just don't feel polished / break rather frequently. So I'm personally stuck with GitHub, simply because no other company provides anything better.

GitLab CI is definitely competitive with GitHub Actions. What's the specific feature set you're missing? "Polish" is pretty vague, especially in the context of a post that laments GitHub Actions' poor uptime record.

Not at all true. GH actions are about as simple as they can be and can be replaced by mostly any other commercial service that does the same. Also, any CI/CD tool can integrate with GH.

Gitlab IMHO is definitely in parity, if not better. I've had loads of success with it at a few places now, using basic stuff right up to full Autodevops with custom buildpacks for Elixir/Pheonix.

I have the exact opposite experience. GitHub CI is probably the worst CI I've used so far (except for custom homegrown messes), and Gitlab CI by far the best.

GitLab CI isn't even on the radar. I'd take Actions or Buildkite anyday over the existing offerings. I'd argue the GitLab CI ecosystem is non-existent.

Wow! I’ve heard GitLab CI is hard to learn because it’s so expansive, but never that it is non-existent. Do you mind sharing more?

GitLab team member here.

I'd recommend starting with one of the workshops listed here [0] and maybe dive into more learning resources and the documentation.

Our community platforms, such as the forum [2] and Stack Overflow, etc. [3] are also a great place to ask questions and collaborate on challenges.

[0] https://about.gitlab.com/handbook/marketing/community-relati...

[1] https://about.gitlab.com/learn/

[2] https://forum.gitlab.com/

[3] https://about.gitlab.com/community/

When I talk to Hashicorp or AWS they provide tooling specifically for GitHub Actions. I don't see the same level of vendor commitment when it comes to GitLab. If I'm not going to get vendor commitment I might as well run buildkite which is awesome or even look to something like Tekton for Kubernetes if I'm just doing Kubernetes anyways.

Gotcha, that makes sense. Thanks for replying.

It's patently wrong there is no community, I've been part of it writing buildpacks.

Wouldn't it be nice, 2 years into these hiccups, if GitHub could explain why the same problem keeps coming up?

There's a point where it's funny, and we're way past that.

They did shed some light into their internal struggles https://github.blog/2022-03-23-an-update-on-recent-service-d...

That's not about Actions though, right? That's about the core mysql for repos.

Actions relies on the main MySQL database, as do many other GitHub services

Again? Last time that happened was 9 days ago. [0] Just like I said before, at least twice a month GitHub Actions, Pages or something else goes down.

Each time this happens, it makes no sense to go all in on GitHub. Perhaps companies like ARM, and projects like Wine [1], ReactOS [2], etc already went with self-hosting or have a failsafe solution to fall back on.

[0] https://news.ycombinator.com/item?id=31815918

[1] https://www.phoronix.com/scan.php?page=news_item&px=Wine-Git...

[2] https://github.com/reactos/reactos#code-mirrors

I think it is not just github actions, github in general is experiencing degraded performance [1].

[1] https://www.githubstatus.com/

Not only is it not more reliable than your own server, using them launders your IP through copilot which they then sell to you and your competitors.

This is axe grinding. Please stop. HN has already had several threads related to Copilot. Take your grievances there, instead.

Hey, does your website accountkiller work for HN? They seem to be rather strict about their accounts.

Source? In other places, it is stated they only use public data, such as from public repositories. See the faq at the bottom here: https://github.com/features/copilot/

That's what I'm looking for. The source - but more so the exact words - that says they're legally allowed to do that. Because even if it's publicly viewable, it's still my intellectual property. I want to read the part that says, "even though it's your IP, we have the right to launder it through copilot and sell it back to you/others".

Emm, sounds not good... this kind of incidents can impact not just developers but also business insight applications being heavily analyzing github activities and projects in real-time, like this one (https://ossinsight.io/).

Anyway to minimize the impact of such github incident on everyone's daily projects and business?

I don't really use github actions (like, ever), but it's the default setup for a terraform provider I'm working on and the very minute I queue a bunch of jobs; the system goes down. Interesting.

The company I currently work for is in the process of migrating out of Jenkins to Github Actions. With all problems that Github has, it has been, by far, a much better experience, even with all these issues and trade-offs, Github Actions in combination with the Github UI has been a net positive in all aspects.

Jenkins is slow and a nightmare to maintain. It became a huge ball of mud that nobody wants to touch. Just keeping the lights on it's a large burden for the infrastructure team.

A bit a false dichotomy: there are other open source CI alternatives that are way more modern than Jenkins: Drone/Woodpecker, Gitlab, sourcehut all come to mind.

My actions are simply not running.

Maybe they need to hire high end talent? Anyone have experience with their recruiting process?

Reply on news.ycombinator.com