Improving how we deploy GitHub
github.blog
github.blog
Does anyone know the pros and cons of GitHub's approach?
The main benefit is, other developers can rely on the master branch even more. They will know there will not be a revert on the master branch they just pulled one hour ago and already started coding on.
Agreed that any codes can be added/removed but those are 100% valid changes in github flow.
EG: If two engineers have two different PRs open modifying `index.html`, how does that go to canary?
This seems like a bit tough for getting work done if you have enough people all waiting around for this
This sounds rather terrifying to me as well. Hopefully there's some sort of system for keeping around all those individual branches that made it out to prod, however briefly, for future debugging/auditing purposes if you ever need them.
It's never fun to have to be doing "what code was running at this time" investigations, but every once in a while it's the only way to really get to the root of something.
This means master is a history of proven stable builds. And in an emergency you don't have to think about what to roll back to, it's by default master.
It's just a convention, it does not have any real benefits or downsides. You could do the same with another branch named stable, or with tags.
> a record of every version that was seen by actual users
This is covered by the release branches and also by the CD pipeline.
Seems one can deploy up to ... Say, 24/3? times per day? If 3 h in between
(trabant00 thanks for explaining :-))
What is the best chatops right now ? I dont see a lot of popularity around chatops. Its most usually some version of github based triggers.
Its funny that Github themselves uses chatops. I think that's a very nice take - especially for early stage startups. Anyone else use anything like it ?
It's missing some of the chatops stuff that is mentioned in the blog post but since we support a lot more languages than Hubot we're hoping it's a matter of time before someone in our community builds a better replacement deployment script (or we'll do it while building out sample scripts :))
(Also, hi GitHub friends!)
To be clear, it's hopefully just some connector that does slack message -> triggers jenkins job.
But from a security, compliance, reliability, debuggability, auditability perspective I think it's inferior. Not to mention an inferior interface.
We liked it because the chat history you see is essentially a deploy history, no need to login into some other website to check some obscure logs page to see who did what. We did end up having to debug the service that processed the chat messages maybe once, but never ran into an issue when we had to deploy a hotfix.
What I do is have all jenkins deploys send a record to the #deploys channel (Service X, version Q deployed by person Y completed successfully in Z minutes), which comes for free with a tiny jenkins plugin.
However one of the unicorns I worked at deleted all slack messages after 3 months for legal reasons, as one example. Also, slack has periodic outages.
I think a lot of people underutilize jenkins, but once you're handy with it (and get over its god-awful ui) you never go back.
Whenever I read comments like this I’m always deeply suspicious of the commenter (is that how you justify trying/adopting tech) or their employer (are they so draconian in tech/design choices that everything is frozen for good). I’m not trying to cast aspersions on you or your employer directly... but it’s fascinating to me to see such a myopic take on a problem space I hope you’d agree is very much not one-approach-fits-all. I’m surprised to hear about their flow too, but my more charitable assumption is that their teams have tried different things and settled on an evolving process that works for them. They’re proud enough to boast it from the corporate blog, it can’t be entirely lark.
Our UAT environment is deployed with fixed versions, and is only updated either after a sprint, or when the business wants to test new features. Generally this is done by someone from the business asking to deploy a new version, and then a developer manually triggering this process. I see no reason as to why the business wouldn't just be able to do this through Slack, and not have developer act as a middleman.
However, if your UAT is a manual refresh with no branch names, for example, that seems perfectly reasonable (so long as it's triggering a pipeline like you mentioned).
However if I worked where you worked and you wanted slack to deploy prod, I'd probably try to talk you out of it.
My annoyance at the moment is that business side will often ask "have we deployed the latest code to UAT?", to which I quickly open Jenkins, check when the latest job ran, and revert back to them. I have tried just linking the Jenkins URL back to them, as to say, "look it up yourself". But I suspect business people just don't want to touch Jenkins, because it's a "technical tool". So my idea has always to been build a simple chat bot, where they can ask when the latest deployment was, and where they'll be able to trigger a new UAT deployment.
> However if I worked where you worked and you wanted slack to deploy prod, I'd probably try to talk you out of it.
We have pretty strict deployment processes for prod, where other teams do the deployments, and it's not even allowed to automate that.
Maybe those who can post to your deployment slack channel, can deploy? So you configure who-may-deploy via permissions in Slack instead?
Here's a relevant talk from 2013: https://youtu.be/NST3u-GjjFw
If cost price is £5 and the markup is 20%, he has to pay £6 to get the part, then charge £7.20 on the invoice to the customer. I’ll let you guess what that does to tender bids ;-)
My uncle’s team was having none of that, so they paid an external computer repair service to fix their computers. The external repair service subcontracted to compaq’s internal people anyway, so when their computers broke they called up (and paid) external consultants. Who in turn called compaq’s internal support team, who came downstairs and fixed their computers at a competitive price.
Consulted for a sub-sub-sub-subsidiary of Toshiba. All computer equipment had to be from Toshiba - the closest place to get Toshiba laptops was two COUNTRIES over.
They even had to tape over non-Toshiba branding from external displays that would be visible.
And Github will definitely still have to "pay" for Teams, whether that is internal accounting or actual money being exchanged.
My guess is that github was using slack before they were bought and inertia is a thing. I'm sure there are people within the parent company that would like to see them transition, but I'm sure there's a ton of resistance, especially "on the ground" at github. Buyouts are a delicate thing, they don't want to ruin github by trying to force it to change too quickly.
More then likely it's because that's what they used before they got bought and haven't been forced to migrate over yet, they also seem to have bots, which are not really a direct copy and paste into MS Teams, and likely them converting over isn't a high priority.
I believe the value of dogfooding would be immense. Not only could you become the customer (massive reduction to the deploy/measure feedback loop), but it would be a key marketing move.
On top of that, the GUI that will now require critical care and development is essentially a clone of what is offered by Github Actions.
It also makes sense internally. Our actions team is much larger than the team that manages the project I wrote/lead, so it also makes logistical sense IMO
We also started before the CD product was ready on actions and we directly influenced it, so yea haha
[0] https://docs.github.com/en/github/collaborating-with-issues-...
> Rebase and merge on GitHub will always update the committer information and create new commit SHAs
This is not a fundamental notion in git. The hash includes timestamps as well.
edit:
It also contains author and commiter details. Here's a slightly revised list borrowed from a blog post[1]
- commit message
- The file changes
- The commit author (and committer- they can be different)
- The date
- The parent commit hash(es)
Not rebase-merging would probably suit your workflow better.
For example, what usually has to happen for a dev to trigger a rollback? Or how do they handle stateful changes such as database schema changes?
That time could very well be 5 minutes but the two need to be coordinated.
Could we reduce risk by lengthening the process? Maybe, but you also make deploys longer which means less stuff can get through in a day. This makes devs respond with larger PRs, for example, which increases the risk profile.
So we need to balance time and duration. Typically large problems will manifest quickly, or take a lot longer to detect (and thus are generally more minor problems) when you have our scale of a user base in my experience.
10-15 is fast I think
Sounds as if you can do more than 100 deployments per day? -- but I guess you don't do that many?