GitHub was having issues | Hacker News Reader

GitHub was having issues

githubstatus.com

121 pointsKiro90 comments

Plug: after almost a full year of hosting alone (for $50) I recently started using srht's builds, somewhat equivalent to github actions. It's been awesome so far. Look at the downloadable coverage.html: super easy to set up!

https://builds.sr.ht/~motiejus/job/517389

And unlike any megacorp, Drew is approachable. I recommend having a look at it for anyone re-evaluating their software forge.

For people using GitHub Actions how often do these types of incidents impact your CI/CD?

Since most CI systems pull code from Github, it impacts other CI providers as well.

Most?

I'd wager that there's more self-hosted CI's running code from self-hosted repositories than from GitHub (however you measure it: e.g., lines of code, build time, etc)

Same here. Self-hosted Jenkins pulling from self-hosted repos seems to be the standard in the companies I worked for and their partners/customers.

All the time. It’s not even that reliable when it’s up. Total embarrassment for us when we have to explain to management.

If I started again I’d probably use something standalone, light weight and 100% self hosted ie GoCD. I’d change the operating model to keep GitHub as a collaboration place but not have any dependencies on it for pipeline. We have other issues as well like the GitHub flow mod being inadequate for maintaining stability on high traffic repos too. I’d pull from master into a production branch and run that off line entirely.

From what I heard it was fine until Microsoft bought it and decided to bring over .net code (and people) from azure devops to replace a bunch of the pre-existing github actions stuff for some reason.

You're saying that .NET is not capable or bad when it comes to handling such a tools like building systems?

I infered that it is more about Microsoft and their management of software development than I did about it being anything technical related.

I read this as more a situation where an existing architecture got stuff shoehorned in for non-technology reasons.

I don’t think there’s a problem with .NET inherently. But swapping out components for new tech stacks for no user or tech reason is likely to introduce bugs and at best will negatively impact schedule.

Although it would be interesting to hear any new company that chose .NET over other alternatives. I really likes how Spolsky explained why they used the MS stack [0] to build stackoverflow, among other things.

[0] https://channel9.msdn.com/Blogs/Seth-Juarez/Joel-Spolsky-Tal...

> GoCD

Don't. Just don't.

I'm assuming that the fundamental flaws* of GoCD haven't been fixed, based on the knowledge that in 2015 they (ThoughtWorks) still hadn't addressed them, and I think 5-6 years isn't long enough for them to be addressed.

*Pipelines, as modeled in GoCD, were so inflexible that any kind of flow resulted in having many, many, pipelines configured. To the point that the number pipelines you operate and maintain is actually the multiple of any fork in logic you need.

Thanks for the heads up. I will avoid that then!

Hate to pile on GoCD. But I second this. In 2015 they were a contender when Jenkins was showing it's age and Gitlab was in its infancy. Development was fairly stagnant and felt 'closed' to external contributions.

I was hopeful, but ultimately disappointed.

Thanks. This sort of stuff is why I come here :)

Very rarely, and (so far) not in any meaningful way. Having to wait a few hours for a CI process isn't a problem 99% of the time. Deploys are affected by issues on AWS and Azure far more often than GH in my experience.

I'd probably change my mind if I ever get unlucky enough to need to deploy a critical hotfix at the same time GH is unavailable though.

If you're OK with waiting multiple hours for CI, I suppose these issues wouldn't affect you very much. Where I work we make relatively small PRs and deploy to production tens of times per day. The recent outages have been a nightmare for us.

> deploy to production tens of times per day

Any chance you're willing to share more?

It sounds like there's something interesting to learn here.

Not the OP but in a continuous release process you treat main (master) as “the source of truth” and merges to main trigger a new release to production.

At the company I work at, we have about 160 full time engineers with about half our repos on continuous release. We average right now somewhere around 50-55 releases per day, with some hot repos getting up towards 20 releases per day on their own.

The point of small releases is that it makes rollbacks super easy (you always know what broke), roll forwards easy (you can easily test post rollout to find out if things aren’t functioning as expected), and you can mentally move on to your next task sooner because your code gets released when you want it released- not when the next “big develop release” happens.

To pull this off requires a few things. You need good tests that run against PRs. You need fast deploys (under 10 minutes is a good starting point, but get them faster if you can). And, ideally, you have ephemeral environments for each PR so that changes can be tested fully in isolation without requiring a shared staging environment.

This is legit what I do for a living, so AMA.

I envy you... my client has a horrible process with lots of manual steps and prod deployment every two weeks.

Can you describe at a high level your development toolset, CI/CD toolchain and workflow, and deployment environment?

Sure thing.

Devs develop locally. Can test their code (manual test and most automated tests).

They push and open a PR. They can deploy ephemeral environments with any dependent microservices spun up along side. This is an in house tool we developed that builds and deploys ephemeral docker, CDK, and terraform based apps.

We then run CI automation against their PRs. This can include E2E tests courtesy of these ephemeral environments.

Once their PR is green and signed off, they merge to master. We’ve done some wizardry behind the scenes so that their code is live in production in around 3 minutes.

They verify their changes in prod and we run any automated regression tests against prod.

Next person merges as they see fit.

That’s it! It’s not easy to get to that point, but once you’re there it’s a great process. We’ll have capacity issues once we get up to 30 or 40 releases per day in a repo, but I’ve got some ideas about how to work around that.

Where I work we make relatively small PRs and deploy to production tens of times per day.

We do that, sort of. Obviously we're not PR'ing to get things in to production. Changes go through QA, Staging, UAT, etc first. The final point is that they go to prod though, and if a few hours of changes end up queued to all go at once that's fine.

If not seeing a change in production for a few hours counts as "a nightmare" for you then I certainly don't envy you. That must be incredibly stressful. How do you do good work under that amount of pressure?

Just about a monthly occurrence now. Harks back to the days of Travis it's frequent downtime.

It's been a weekly occurance for months now.

What's the reason these Git services have so many issues all the time?

possibly Azure?

Doesn't look like it, unless the status-page update is also affected :)

https://status.azure.com/en-us/status

It seems that status pages never work properly in those kind of situations. The only reliable source of information seems to be twitter and HN

Saying that twitter is the only source of truth for something usually means that this something is utterly broken... :D

...or you're in the mind of your user.

What if Twitter has an outage...

Reminds me of the time when Twitter had an outage, because an FB outage made people go on Twitter to ask whether FB is down

As far as I know GitHub doesn't use Azure at all, I think it's still hosted by Carpathia. https://github.com/holman/ama/issues/553

For github actions .net code and people were brought over from azure devops to replace existing stuff when MS bought it. At least so i've heard

Actions is entirely on Azure.

The hosted runners are, the backend and core isn't

Aren't these pretty heavy duty and any issue sticks very prominently ?

A ton of companies are using these tools literally 24/7, at most of my jobs Github/Gitlab being down meant most of us would just be spending half of our time faking working (rare are the people who could just do "offline" dev for half a day without any external input/output)

Slack could be down it had less impact overall.

You could use that downtime to learn about the variety of ways that git natively supports distributed workflows [1]

[1] https://git-scm.com/book/en/v2/Distributed-Git-Distributed-W...

> We have discovered the source of the errors and have just rolled out a potential fix. We are now monitoring recovery and will update soon.

Looks like some code issue from their side.

Just like every company, they sometimes have things break, push out code that doesn't behave as expected, hit limits on various services they weren't aware of, etc.

The complexity of technical systems is massive especially when they are constantly changing. It's not helped that these companies have high availability requirements and need to be online 100% of the time even when making updates.

Makes me want to do embedded programming.

I love github as a product but for the past months, issues are a weekly occurance, often multiple times a week, and always in the middle of the day (Europe).

GitHub were pretty open about having issues with Bitcoin mining abusing the free compute available via Actions. I think it's an ongoing battle and one which is having a wider platform impact

https://status.circleci.com/ CircleCi having issue due to this

Pretty much any CI provider will be "having issues", as most of the time builds are triggered by GitHub webhooks (and also pull code from GH)

Things like this make me want to have more free time to start my own paid CI service. I can likely start from a few languages I know and take it from there.

I wonder how hard it is to write your own CI/CD though? Anybody has thoughts on this?

If you Google around you’ll see a crap ton of companies trying to do this.

IMO the only “serious” threat to Jenkins are platform integrated services like Gitlab CI/CD, GitHub Actions, Google CloudBuild etc. Everybody else seems to continue to use Jenkins.

Good to know, thank you.

I'm not scared of competition, I'm mostly wondering how viable and profitable such a business would be. I happen to believe people would pay for good specialized CI (say, Elixir-only, Rust-only, Golang-only etc.) but you're also quite right that the real value-add seems to be in integration.

As long as you don't provide a huge degree of additional value you will be more likely to lose money through free plans which are needed to onboard people then you are to make money.

While specialized CI sounds great you would need a good general purpose CI which also sports specialization. Ort people will need to change CI once they e.g. need to build and link against some FFI. Or have some project in another language. Which is quite a bit reason not to just it even if you currently only need specialized CI.

I recommend looking at buildbot[1]. It can be tailored towards your needs. I developed and maintain one for scummvm [2].

It has a steep learning curve, as with any sufficiently advanced system.

[1] https://buildbot.net/ [2] https://john.scummvm.org

I would bet on it being related to their identity server. It seems to almost work in browser(s) where I was signed on - but in a new browser, it gives HTTP 500 after SSO.

Can confirm. I almost go crazy a few minutes ago when I pushed some commits and couldn't see them appearing in the Pull Requests

It was also impacting acting raw gists which is not noted in their report.

Does it affect self-hosted runners as well?

We're looking at switching to self-hosted runners, so I'd be very interested in hearing if it would shield us from some of Github downtime.

We're also looking into Github enterprise for the IP allowlist feature which unfortunately means we'll have to migrate our whole CI/CD to self-hosted since Github-hosted runners don't work with this feature enabled.

Maybe these are just a bunch of coincidences, but I have a feeling that GitHub is being poorly run recently. Apart from quite frequent uptime issues, it seems that Github Actions is pretty half-baked and they completely botched the migration of Dependabot (for example, take a look at this issue and especially, how it was handled: https://github.com/dependabot/dependabot-core/issues/3253)

A shame they removed all their uptime graphs on the status page, since it's much harder to get a read on things now.

However, scrolling back through the status page's history for the last year shows that there between 6 and 12 incidents every month, and it often hovers at the high end of that range.

That has to be an utter misery for anyone who's on call.

> A shame they removed all their uptime graphs on the status page, since it's much harder to get a read on things now.

I've started using that (uptime/performance/latency graphs being removed from status pages) as a signal to for when companies feel embarrassed over their quality of service but don't have enough interest/resources/profits to fix it.

> interest/resources/profits

It doesn't seem like Microsoft should have any of these issues. At most "interest", but even that should be a pretty easy sell for a service as massive as github.

As Microsoft has SLAs, it probably isn't in their interest to make it easy to tell, to begin with. Management would probably just see liability.

Makes it much harder for enterprise customers to claim on their SLA as well.

...so that customers start tracking it themselves, so that everyone can waste time on discussing their differences in numbers and measurements, and who is right and who is wrong.

So who's ACTUALLY got the best reliability?

They have also largely abandoned their Terraform and Jira integrations.

Has been left up to the open source community to implement basic functionality.

OT but why did it capitalize the title like that? I submitted "GitHub is having issues". I don't see that for other submissions (unless people manually edit them afterwards).

EDIT: Before the change to "GitHub was having issues" it had converted my original submission from "GitHub is having issues" to "GitHub Is Having Issues".

I first interpreted it as a puny way to advertise some improved github issue tracker

There are some rules for when this happens and when it does not. Don't know what they are exactly. Maybe the camelcase in GitHub may have triggered. Anyway, as you say, afterwards you can edit the title and give the desired case format.

From what I can tell from looking at older submissions, it might be doing it if the title is 4 words or less. Not sure why that would be the case though..

Too much smarts in the wrong places imo..

It is title case: https://en.wikipedia.org/wiki/Title_case

I also find myself writing titles in a "Title Case" like that. It is a habit I got from the days when I used to blog.

BTW you can see changes that occur like this on https://hackernewstitles.netlify.app/ - it seems like the automated title casing is being overridden by mods later on, perhaps.

Just found this out now, which is why I decided to submit my current blog post as it is :D But I'll finish it soon enough eheh

Here we go again [0] and again [1] and again [2] and again [3].

Time to reset the counter once again. Now everything there is degraded performance. Oh dear.

Consider a self-hosted alternative or have one as a backup like I have said before.

[0] https://news.ycombinator.com/item?id=27192869

[1] https://news.ycombinator.com/item?id=27172443

[2] https://news.ycombinator.com/item?id=26666843

[3] https://news.ycombinator.com/item?id=26667101

Cue the age old argument chain about outsourcing responsibility vs having control of downtimes.

I’m tired of them now, anyone using hosted solutions should know this by now and they’ve made that choice.

Personally I believe people underestimate how easy it is to host your own services.

But I’m sick of having the conversation over and over, if people aren’t willing to investigate then, what’s the saying: “you can lead a horse to water but you can’t make them drink”

But the argument isn't static. GitHub has become much more of a target to large-scale attacks as it's grown in popularity and has become mission critical to so many companies and projects.

Security risk isn't a one-and-done decision. It's a changing landscape and, arguably, there are far too many eggs in the GitHub basket for this not to be something people might want to think twice about now.

GitHub also arguably just has a good product that's a good fit for a lot of use-cases, regardless of if there's better options.

Yeah, you might be able to self-host gitlab/gitea/etc but their free offering for organizations is so good that for my 30 commits/week open source project, I can deal with half an hour of downtime once in a while.

Combined with free hosting, it's really not that much of a stretch that people are willing to put up with this, especially as there have been periods of time with less outages before, though that's been a while now.

> especially as there have been periods of time with less outages before, though that's been a while now.

Github hasn't been reliable for a while now.

P.S. Most people here attribute being a "hacker" to curiosity.

self hosted won't insulate you from your VCS instance failing. I used to work at a medium-sized company with Gitlab 8 years ago, it was running into several issues and required frequent maintenance.

To give another data point, in my company we're running Gitlab (we started 3,4 years ago maybe?) with some pretty heavy use of CI and we've had no issues, even with automatic upgrades enabled (we have tested backups, we're not crazy).

Confirmed, on a ubuntu host GitLab is actually the product that I have the best self hosted experience with. Apt packages running their Chef Cookbooks always impressive!

Well going all in on GitHub / GitHub Actions now really sounds like a bad idea. Even some others had issues [0] and had no external backup plan to counter GitHub's regular outages.

The best example to learn from is ReactOS who while has moved to GitHub they at least have a self-hosted backup just in case GitHub falls over. [1]

For the ones who are not on GitHub and are self-hosting (OpenBSD, Mozilla, Linux Kernel Project, GNOME, xfce, wireguard) I don't hear any complaints from them.

[0] https://news.ycombinator.com/item?id=26301523

[1] https://github.com/reactos/reactos#code-mirrors

Is there anything at in the same league as https://reviewable.io for code reviews on self-hosted Git?

Theres gerrit and reviewboard, both of which I would consider quite nice.

I dont know reviewable so it might have more sauce than the recommendations

The UI is much nicer than gerrit (or at least the gerrit UI I knew from 5-6 years ago not sure if they have changed). It’s more “GitHub”-y. It’s a bit confusing at first but once you get into it it’s pretty damn powerful.

Side note, I honestly seriously think we need much better tooling than just these 2 for code reviews.

Gerrit has gotten a new UI in the last few years as far as i know. I wouldnt nesecarily describe it as pretty, but its my personal favourite review system

I'm building a much better code review for tool for GitHub, check out https://codeapprove.com

It combines concepts from GitHub, Gerrit, and other good review tools out there to make it easy and fast to reach consensus (which is what it's all about).

If you want to hear more you can email me at sam at habosa dot com

> Consider self-hosted alternatives like I have said before.

This assumes that your self-hosted Git service will have better uptime than Github.

> This assumes that your self-hosted Git service will have better uptime than Github.

Piece of cake. But it will also have better performance.

Not even that necessarily, control of downtime is also worth something. i.e. lots of downtime is tied to system changes/updates, especially in a conservative setup, which you can schedule as appropriate on a self-hosted setup.

Ye ... I mean most messups happen when stuff is changed, which you can choose to not to do on a release day etc.

Someone said the same thing to me more than a year ago [0] and here we are the same issues once again.

This is why it makes no sense to go 'all in' or try to "centralize everything to GitHub". At least (if you're a company or an organisation) have a self-hosted backup and don't jump into the 'goin all in' hype train.

[0] https://news.ycombinator.com/item?id=22868406

> This is why it makes no sense to go 'all in'...

I completely agree with you, to reiterate what I said last time we had this discussion 3 months ago:

> I wouldn't call it prudent to go all-in on any cloud service [0]

I've never used Github as anything more than a managed git service with a nice web interface. At work, any time the topic of moving our CI/wiki/project management/issue tracking onto GitHub has come up, I've been vocally opposed.

If GitHub went down, and didn't come back up, it would be trivial to migrate to Gitlab/CodeCommit/Bitbucket. The most time consuming thing would be setting up new web hooks for our CI.

[0] https://news.ycombinator.com/item?id=26302232

Reply on news.ycombinator.com