Is this really beneficial? Has there been any meaningful analysis of the use of CI/CD - in parts or in whole?
My limited experience with CI/CD has shown significant overhead, both human and compute, but much less end user value. Aside from research, I would enjoy hearing both positive and negative experiences people here have had.
They can be taken as a signal that the deployment process is safe, automated, and low-overhead. When that applies, I have generally seen the number of deployments go up. The reason is pretty simple: engineers stop worrying about deployments, the overhead goes away, and deployment becomes part of the "definition of done".
My experience is that infrequent integration and deployment is a more painful process when it comes to integration and deployment time. Easy deployments encourage features and bug fixes to be rolled out as atomic units when they are completed, rather than saving them for big-bang releases. This encourages small, independent changes, rather than larger ones that involve more simultaneous code and data model changes – often in my experience multiplying the chance of failure by an outsized factor.
Honestly I'd actually go in entirely the opposite direction and say that one of the most valuable and beneficial systems I see in companies is an effective CI/CD system with frequent deploys. I find it discourages bad practices and encourages good ones; I would consider it a red flag when looking to work with a company if they didn't have such a process in place.
Deployments should be quick, simple, reliable and accessible to developers. With automated builds + unit tests you can have simple deployments that take < 10 minutes if not < 5 minutes without too much complexity on most medium sized projects.
This lets you ship often and break complex behaviour into smaller tasks. Exposure to bugs is limited to only what's shipped, so you can be confident that each deployment is correct.
I think the idea that there is significant overhead seems to me to be the opposite - frequent small deployments are simple and easy to verify. Single large deployments often change huge amounts of code, multiple DB migrations, and introduce multiple opportunities to fail.
If you deploy too infrequently, you can also develop a culture of rushing to get into the release, or being (or your customers being) scared to push to production due to bad experiences in the past.
My company has embraced CI/CD. Unfortunately, they have created metrics around it. In some instances this has created headaches because managers want us to do stuff to make the metrics look good. I've seen the same thing with other practices in our company, such as the story points in Agile.
When they list it as positive bullet point they are saying you are not going to be chasing "Jimmy the only guy that can do deploys" to get your work deployed and so on.
There are lots of crappy teams that deploy lots of times a day too though so keep that in mind as well.
In that case, you quickly get update fatigue from the user. From experience, every couple of months or so is a good trade-off.