Good thing git was designed as a decentralized revision control system, so you don’t really need GitHub. It’s just a nice convenience
The downtime we do have each year is typically also on our terms, not in the middle of a work day or at a critical moment.
The reason for buying centralized cloud solutions is not uptime, it's to safe the headache of developing and maintaining the thing.
Multi-AZ RDS is 100% higher availability than me managing something.
Anecdotal, but ¯\_(ツ)_/¯
Most software doesn’t need to be distributed. But it’s the growth paradigm where we build everything on principles that can scale to world-wide low-latency accessibility.
A UNIX pipe gets replaced with a $1200/mo. maximum IOPS RDS channel, bandwidth not included in price. Vendor lock-in guaranteed.
Meaning the cloud may go down more frequently than small scale self deployments , however downtimes are always on average much shorter on cloud. A lot of money is at stake for clouds providers, so GitHub et al have the resources to put to fix a problem compared to you or me when self hosting.
On the other hand when things go down self hosted, it is far more difficult or expensive to have on call engineers who can actual restore services quickly .
The skill to understand and fix a problem is limited so it takes longer for semi skilled talent to do so, while the failure modes are simpler but not simple.
The skill difference between setting up something locally that works and something works reliably is vastly different. The talent with the latter are scarce to find or retain .
I’ve seen that work first hand to keep critical stuff deployable through several CI outages, and also has the upside of making it trivial to debug “CI issues”, since it’s trivial to run the same target locally
You should aim for this but there are some things that CI can do that you can't do on your own machine, for example running jobs on multiple operating systems/architectures. You also need to use CI to block PRs from merging until it passes, and for merge queues/trains to prevent races.
Ended up expanding this little quip into a blogpost to refer to in the future, feedback welcome! https://tech.davis-hansson.com/p/ci-offgrid/
We moved to GHA b/c nobody ever got fired ^W^W^W^W leadership thought eng running CI was not a good use of eng time. (Without much question into how much time was actually spent on it… which was pretty close to none. Self-hosted stuff has high initial cost for the setup … and then just kinda runs.)
Ironically, one of our self-hosted CI outages was caused by Azure — we have to get VMs from somewhere, and Azure … simply ran out. We had to swap to a different AZ to merely get compute.
The big upside to a self-hosted solution is that when stuff breaks, you can hold someone over the fire. (Above, that would be me, unfortunately.) With Github? Nobody really cares unless it is so big, and so severe, that they're more or less forced to, and even then, the response is usually lackluster.
It just workz [;
I am fairly certain that the vast majority comes from improper use (bypassing security measures, like riding on top of the cabin) or something going wrong during maintenance.
Maybe I've just been unlucky, but so far my experience with CI pipelines that have extra steps in them for compliance reasons is that they are full of actual security problems (like curl | bash, or like how you can poison a CircleCI cache using a branch nobody reviewed and pick up the poisoned dependency on a branch which was reviewed but didn't contain the poison).
Plus, it's a high value target with an elevated threat model. Far more likely to be attacked than each separate dev machine. Plus, a motivated user might build the software themselves out of paranoia, but they're unlikely to securely self host all the infra necessary to also run it through CI.
If we want it to be secure, the automation you're talking about needs to runnable as part of a local build with tightly controlled inputs and deterministic output, otherwise it breaks the chain of trust between user and developer by being a hop in the middle which is more about a pinky promise and less about something you can verify.
It would be great to also have the continuous build and test and whatever else you “need” to keep the project going as local alternatives as well. Of course.
[1] Or maybe there is just that much downtime on GitHub now that it can’t be shrugged off
It's more work and slower. I'm convinced half of the reason they keep it that way is because the barrier to entry is higher and it scares contributors away.
You mean, assuming everyone in the conversation is using different email providers. (ie. Not the company wide one, and not gmail... I think that covers 90% of all email accounts in the company...)
You can commit, branch, tag, merge, etc and be just fine.
Now, if you want to share that work, you have to push.
Yes you lose some convenience (like GitHub's pull requests UI can't be used, but you can temporarily use the other Git server's UI for that.
I think their point was that you're not fully locked in to GitHub. You have the repo locally and can mirror it on any Git remote.
It is awfully convenient, web interface, per branch permissions and such.
But you can choose a different server.
Well you can with some effort. But there's certainly some inconvenience.
But yes ssh pushing was down, was my first clue.
My work laptop had just been rebooted (it froze...) and the CPU was pegged by security software doing a scan (insert :clown: emoji), so I just wandered over to HN and learned of the outage at that point :)
However, since we use github.com fore more than just a git hosting it is SPOF in most cases, and we treat it as a snow day.