GitHub Actions Down
And I'm here waiting for my scheduled build to start...
I now doubt if they can consistently manage more than a month without a major incident like this one.
IIRC, GitHub, Bitbucket and GitLab.com are all unusable for a few hours at least once a month as far back as I can remember.
Isn't this just commonly accepted as a tradeoff for not having to manage the servers yourself?
Stability & reliability are relatively easy to achieve if you aren't changing the software frequently.
Companies and programmers should be aware what they get into when they build such dependencies. Distributed git is a thing, but distributed CI/CD that you could also run locally isn't (yet?).
That’s the part I like about self hosting. It may ultimately be down more often, but I never have to tell someone “Nothing to be done, we wait.”.
What's better, on the other hand, is that you can schedule expected and possible downtimes to a time that causes the least impact to your company; with a SaaS, an update might cause you problems any time.
In these kinds of situations, you often end up in a situation where they say ‘the problem is resolving itself in region x’, where region x is not relevant to you at all. If you are fixing your own setup you can focus on exactly what is most important (to you) first.
You won't get high availability from Microsoft you are used to as from proper cloud services. Plus privacy issues. But it's cheap, in this case for free.
Edit: Here it a tip; if you see a "Microsoft partner <TIER> Cloud Platform" badge on a outsourcers website stay away.
The worst part about Azure for me is always the list of undocumented bugs you run into. On the surface it looks like everything started as an AWS equivalent, but when you have to drill down on something it almost always has some weird issues that you then find as unresolved complaints on some MS managed github issue list.
But hey, maybe I was just luckier with the other cloud providers.
This and other API weirdness gave me such Azure PTSD that I promised myself I would never touch it again.
Not a single one i could manage building ( eg. Teams has bots, quick to create apps and pretty advanced cam features)
Every time I tried azure I was disappointed. But that doesn't mean they can't fix it; I bet there are now tons of talented engineers working there. My best wishes for them to up the quality of azure. I think diversity / alternatives are a good thing.
As unreliable as GitHub Actions are, their convenience factor and price are right.
We take a very simple measure so we don't get fucked by these kinds of incidents: we don't use any actions from the marketplace.
All our GitHub Actions workflows are bash scripts that we wrote (and which often live in our repos at `deploy/deploy.bash`). The secrets necessary to run these scripts are available to the infrastructure team on 1Password.
This makes it easy for us to deploy manually and retroactively reflect that release on GitHub (e.g. through a tag or a release).
[0] https://github.community/t/github-action-stuck-on-starting-w...
The portability could be fixed by having a local cli runner that understand action yamls. Would be interesting to explore this.
For a while i have been fantasizing about an universal pipeline language. Like having LLVM with a unified model that can translate into different vendor implementations.
This is a great idea. I guess the challenge would be keeping up with the more advanced aspects of Actions, eg. spinning up multiple VMs during the build process and other things that have a heavy infrastructure (or platform-specific) element.
I'm currently caught by the GitHub Actions downtime, but like the GP, all my build scripts generally make very light use of the platform-specific features and generally I keep most of the build logic in a build.sh file.
So I can build production builds locally if I need to - but a CLI that lets you "properly" run the GitHub yml files locally would be very interesting.
By the way, should GitHub send an email to the owner if any workflow has been delayed for an unreasonable time?
The question was do they do a email if your job is delayed or late for whatever reason?
Not, hey why don't they email us all right now about the issue.
And no, it's not on me to monitor every little thing I rely on. Do you monitor kernel updates? I bet you don't. Besides that monitoring and logging for any provided service is exactly how one is supposed to monitor said external services so asking about monitoring options and being told, look buddy it's your job to monitor for this is just fucking rude.
The internet fails to connect when running yarn install about 3 out of every 5 times.
We’ve gotten multiple refunds for this issue, but it’s still a complete mess. Unfortunately we’ve built much of our process around GH Actions... if not I’m sure we would be on CircleCI or TravisCI already. We’ve also considered switching to self-hosted runners.
We get the advantages of the huge GitHub Actions ecosystem while having impressive (and fully controllable) performance and very easy access to out infrastructure for deployments.
We're currently using an Intel Mac we've used from before we migrated from Jenkins to GitHub actions, so your mileage may vary.
Having the ability to build & deploy your software outside the confines of a cloud vendor is essential to survival. When the automation works, its great. When it doesn't, have a manual process that you can follow on a local workstation.
At the end of the day, you can always email the customer a zip file and walk them through installing the update in production. That is, as long as you didn't make your architecture and CI/CD one in the same thing, in which case you probably need to hit the reset button and try again.
If you know how to do things like Process.Start, there's really no excuse for not being able to automate your build processes using code. MSBuild has a pretty damn simple set of CLI args if you are just doing modern .NET 3.x/5.x apps.
git clone <my repo path>
cd <my repo path>
dotnet build --configuration Release
//copy build artifacts to where ever they need to go
We use SQLite, so there aren't any dependencies outside of any particular checkout of the repo.
https://docs.github.com/en/actions/hosting-your-own-runners/...