For people using GitHub Actions how often do these types of incidents impact your CI/CD?
If I started again I’d probably use something standalone, light weight and 100% self hosted ie GoCD. I’d change the operating model to keep GitHub as a collaboration place but not have any dependencies on it for pipeline. We have other issues as well like the GitHub flow mod being inadequate for maintaining stability on high traffic repos too. I’d pull from master into a production branch and run that off line entirely.
I don’t think there’s a problem with .NET inherently. But swapping out components for new tech stacks for no user or tech reason is likely to introduce bugs and at best will negatively impact schedule.
Although it would be interesting to hear any new company that chose .NET over other alternatives. I really likes how Spolsky explained why they used the MS stack [0] to build stackoverflow, among other things.
[0] https://channel9.msdn.com/Blogs/Seth-Juarez/Joel-Spolsky-Tal...
Don't. Just don't.
I'm assuming that the fundamental flaws* of GoCD haven't been fixed, based on the knowledge that in 2015 they (ThoughtWorks) still hadn't addressed them, and I think 5-6 years isn't long enough for them to be addressed.
*Pipelines, as modeled in GoCD, were so inflexible that any kind of flow resulted in having many, many, pipelines configured. To the point that the number pipelines you operate and maintain is actually the multiple of any fork in logic you need.
I was hopeful, but ultimately disappointed.
I'd probably change my mind if I ever get unlucky enough to need to deploy a critical hotfix at the same time GH is unavailable though.
Any chance you're willing to share more?
It sounds like there's something interesting to learn here.
At the company I work at, we have about 160 full time engineers with about half our repos on continuous release. We average right now somewhere around 50-55 releases per day, with some hot repos getting up towards 20 releases per day on their own.
The point of small releases is that it makes rollbacks super easy (you always know what broke), roll forwards easy (you can easily test post rollout to find out if things aren’t functioning as expected), and you can mentally move on to your next task sooner because your code gets released when you want it released- not when the next “big develop release” happens.
To pull this off requires a few things. You need good tests that run against PRs. You need fast deploys (under 10 minutes is a good starting point, but get them faster if you can). And, ideally, you have ephemeral environments for each PR so that changes can be tested fully in isolation without requiring a shared staging environment.
This is legit what I do for a living, so AMA.
Can you describe at a high level your development toolset, CI/CD toolchain and workflow, and deployment environment?
Devs develop locally. Can test their code (manual test and most automated tests).
They push and open a PR. They can deploy ephemeral environments with any dependent microservices spun up along side. This is an in house tool we developed that builds and deploys ephemeral docker, CDK, and terraform based apps.
We then run CI automation against their PRs. This can include E2E tests courtesy of these ephemeral environments.
Once their PR is green and signed off, they merge to master. We’ve done some wizardry behind the scenes so that their code is live in production in around 3 minutes.
They verify their changes in prod and we run any automated regression tests against prod.
Next person merges as they see fit.
That’s it! It’s not easy to get to that point, but once you’re there it’s a great process. We’ll have capacity issues once we get up to 30 or 40 releases per day in a repo, but I’ve got some ideas about how to work around that.
We do that, sort of. Obviously we're not PR'ing to get things in to production. Changes go through QA, Staging, UAT, etc first. The final point is that they go to prod though, and if a few hours of changes end up queued to all go at once that's fine.
If not seeing a change in production for a few hours counts as "a nightmare" for you then I certainly don't envy you. That must be incredibly stressful. How do you do good work under that amount of pressure?
I'd wager that there's more self-hosted CI's running code from self-hosted repositories than from GitHub (however you measure it: e.g., lines of code, build time, etc)