Hacker News Reader

I kind of disagree. While obviously these three things (VCS, CI, and Deployments) are different things, there's a lot of value in reducing all of them to just one thing: VCS is truth. It can make the system far easier to reason about.

Question 1: What's running in production? Allowing production to run "anything" means "anything" could be running there, so you need tooling to answer that question. Maybe that's just "give every developer read-level access to AWS", or maybe its a slack slash command, or something, but there's a cost there. With a coordinated model: its just the main branch. Its always the main branch. So look at the main branch.

(Sidenote: this isn't a perfect solution, because oftentimes there's a temporal delay between a main branch merge and deployment, and developers want to know precisely when the cutover starts/finishes. So you end up building that extra stuff anyway. But: that can be delayed quite a bit, as its a higher level question given that the main-is-production model solves a lot of problems alone).

Question 2: How do I re-run the build? Well, you can push a new commit, or issue some slack slash command, or maybe a comment in the PR that reads `!buildbot run` or whatever. How about we just make it simpler? Just push a new commit.

The broader point is that: turing machines can do anything. Human brains can't. So when operations are simplified around models like "source-is-truth" its because that model is a satisfying fit for the problem space which helps the human brain reason about what is happening. All models are wrong; but some are useful.

> VCS is truth

VCS can only ever be intent at best.

I agree. This goes back to the recent post "don't use latest". If you don't pin everything you can't claim that VCS is truth. A built binary is the truth, whether that be a container image, go bin or whatever. Even then I wouldn't say it's the absolute truth. The whole system is the truth.

"VCS is truth" does not assert what it is, but rather its treatment in the system. There is no knowable truth in large-scale systems design.

Velocity is relative in physics, right? Relative to what? What's the frame of reference? Usually, we say, the Earth itself. VCS is that to software engineering, in this setup; there's movement and realities outside of VCS, but VCS is our zero; the tare. Everything is relative to the VCS. It is not truth through its nature; it is truth through how we treat it, how we build systems to interface with it, synchronize with it, etc.

All models are wrong, but some are useful.

> Allowing production to run "anything" means "anything" could be running there, so you need tooling to answer that question. Maybe that's just "give every developer read-level access to AWS",

You can setup your deployment tools to always deploy the lastest from a specific branch unless manually overridden. It is also not very difficult to post the deployed got commit hash to an internal channel, or even expose it via a web interface. Node of this require designing your system so that the only way to trigger a new deployment is by pushing a new commit.

> Well, you can push a new commit, or issue some slack slash command, or maybe a comment in the PR that reads `!buildbot run` or whatever. How about we just make it simpler? Just push a new commit.

And what happens if you want to test building before your change is fully ready to merge or roll back to an older release? It is better to design a modular system than to try to hack this basic stuff in when you suddenly need it.

Then there's my security concerns of conflating "write access to github" full access to the environment. If you allow code to be deployed immediately on commit, then anyone who can commit can compromise your environment.

> If the build fails, you should be able to re-run it without pushing to version control.

I've uh, not seen one that doesn't? (I agree with you, it shouldn't. Does your CI system not have a "retry" or "restart" button?)

> The most shocking of all, though, and I have actually seen this: If you deploy a broken build, a rollback to an earlier version should never, ever, require a rollback on your source control.

So, my company does this, and I love it; it gives even the least skilled of our devs a good shot at the "recognize things aren't working → problem started with last deploy → revert last deploy" debug path, and getting it right. Having that last step of revert a deploy literally correspond to a git revert is very useful & easy.

Keeping the "state of what is deployed" in code also gives us good confidence that there aren't random changes being made, allows for code review, provides a log of who changed what when — all the goodness of infrastructure as code.

Now, we keep the main production "monorepo" separate from our repo that describes what is deployed (let's call it "deploy-state". So a revert of a deployment happens in the "deploy-state" repo, not in the monorepo. So, someone's merged feature branch isn't going to get reverted. (Though, we should have a conversation about whatever the actual problem is: if it is code in the monorepo, why did it not fail tests there? We also have ways of saying "this range of commits, from [break, fix), should not get deployed to prevent repeats.)

Whether that setup is a "true monorepo" or not, since it literally involves more than one repo … well, you be the judge. It seemed to us pragmatic to have it be separate, and we've not regretted that.

This isn't that very different from having a "deploy-state" branch that runs alongside your master branch too, bringing it even closer to a true monorepo.

> So a revert of a deployment happens in the "deploy-state" repo, not in the monorepo.

This makes me think you're disagreeing with the literal interpretation of what I wrote, while very much agreeing in spirit.

I'm not trying to say that a VCS is an intrinsically wrong solution to managing deployment state (I'm not a fan, but it's an eminently reasonable approach). My issue is with deployment state and application source being enmeshed in the same repo such that you can't evolve one without messing with the other.

> Does your CI system not have a "retry" or "restart" button?

It's pretty common on GitHub that PRs trigger a build but unless the author has some permission on the target project already, can't trigger a rebuild. I'd estimate 30%+ of the PRs I create fail for reasons unrelated to my changes and I have to ask someone to retry.

Ah, if the project is open-source & its CI is external, I guess I could see that. I was reading the remark in the context of "within a company", where devs would have access to the CI system (& thus, the "restart" button).

Another class of problem, even without errors, is needlessly restricted concurrent access: your teams tries to push an urgent fix that needs to be built, tested and deployed an hour ago, to find that the CI system will be on its knees, or actually locked, because another team is compiling and testing something unimportant. In many cases, insult can be added to injury by forcing you to waste more time after the other build finishes to change version numbers or merge completely irrelevant changes in source control.

We have reverts bypass testing and most of CI. A revert commit on master goes directly to production as fast as possible. Nothing should stop an emergency revert. If it’s not an emergency, it goes through the normal process.

Don't all action runners allow you to re-run tests and builds with a click of a button and most platforms allow rolling back to any arbitrary commit number?

I have never seen any of the above issues.

(also I don't agree on not rolling back, I want one branch to always be the actual code living on prod, or else you end up with arbitrary time wasting rabbitholes of accidentally debugging code that isn't live)

So you're opposed to practice of canary deployments?

> If you deploy a broken build, a rollback to an earlier version should never, ever, require a rollback on your source control.

Why not? In some cases in lighter weight systems, you might have a release branch that gets pushed to production. Like dokku.

Key word there is "require". Having that automation set up is super convenient and a pretty important factor in making things more productive, not contesting that.

The other side of the equation is that if I break something, I want to make the smallest change possible that fixes it, so as to avoid confounding variables if it doesn't work. Off the top of my head:

  * I don't want to make things worse by messing up the rollback due to stress
  * I don't want to trigger a build and wait for CI to run and build
  * I don't want to depend on my build not being perfectly reproducible (e.g. because the problem comes from a bad dependency upgrade and the rollback rebuilds with the bad dependency)

What I do want is a really simple way to just tell my system "You know that thing you were running 5 minutes ago? Please go back to running that again".

Thank you! I thought I was alone in thinking like this.

I hate Atlassian Bamboo with a passion but it is the only CI/CD system I remember coming across that actually has a concept of deploying a built artifact.

Have you tried GoCD?

I haven't yet. Good to know there are options. Thanks!