Poll: How often do you deploy to production?
How often do you deploy new versions of your code to your production server? Manually or using CI, including quick bug fixes and patches.
If it's a critical fix, then it goes out asap. If it's a minor fix, but relating to a new feature roll-out from 2 hours ago, it too goes out as soon as it's ready. Otherwise, changes gets deployed once the feature is complete.
I wouldn't recommend this set up. I am incredibly competent. I make maybe 1 minor mistake a month - if that - so with me this set up is workable. The other developers I work with.. not so much. The main issue with this set up is newbie developers deploying what they thing is a good fix which breaks in lots of edge cases. Bad patches like that can take weeks to become apparent then longer to chase out of the system as there is no way to simply rollback.
For my side project I deploy every Wednesday. Every now and again there is a bug fix which needs to be slipped in on an odd day. I currently deploy manually because I haven't set anything else up.
Throw up maintenance page -> update database -> upload new files -> quick test -> remove maintenance page.
It is a bit time intensive but I haven't had any problems with it yet. I like to group changes so I can announce them on the same day.
What type of product do you have? Do you have a QA team? Do you track new and close bug counts from week to week? Similarly do you track regression counts from build to build? Does the code base have a high completeness rate for unit tests? Do you only deploy if you pass the unit tests? What if you introduce a bug that screws up the data, do you have a fallback strategy?
I'm curious because I always had a dream to be able to deploy on a daily basis but I've never gotten there "yet". It's always nice to hear what others are doing for daily deployments.
Honestly, I don't know where "incredibly" came from. I was going for pretty (pretty competent). I think my mind was in 2 places at once.
I have been working with the same code base for 2 years. I know it back to front. I am good at testing and my mind is open to the various scenarios which need to be tested for each change which is made. I make mistakes all the time but they very very very rarely sneak out and get live.
> What type of product do you have?
Its a website-as-a-service product.
> Do you have a QA team?
Nope. For major features we try to get all employee's to pitch in for a group test on a development server. That is the best we do. Minor features / fixes are just tested locally by the developer then go out.
> Do you track new and close bug counts from week to week?
We use bug tracking software. We track the amount of open bugs. When I started I was getting around 20 bugs / developer support requests a week. Now I get one or two. Most of what I get through the tracker is trouble shooting unfamiliar hosting configurations.
> Similarly do you track regression counts from build to build?
No
> Does the code base have a high completeness rate for unit tests?
The code base doesn't have unit tests. It was built by a hobbiest 4 years ago. The current version supports legacy features from a previous version. It is mainly procedural with a smattering of random classes. There is a lot of code duplication. It is the definition of spaghetti code.
Basically, the core software needs to be rewritten. This isn't a management priority so we make do.
> Do you only deploy if you pass the unit tests?
N/A
> What if you introduce a bug that screws up the data, do you have a fallback strategy?
We have twice daily backups and can rollback.
1. Why bother with two SVN repos? 2. Any specific reason for SVN at all (large files, binary files, non-technical users)
2. It was what someone chose 4 years ago. The guy has since left. Everyone is familiar with SVN. SVN is pretty easy to get into. We haven't seen a need to change.
The code base doesn't have unit tests. It was built by a hobbiest 4 years ago. The current version supports legacy features from a previous version. It is mainly procedural with a smattering of random classes. There is a lot of code duplication. It is the definition of spaghetti code.
Honestly I wouldn't know where to begin with unit tests in this code base.
I'd wager many people here missed "the good old days" when it wasn't uncommon to debug a live website over ftp.
But, when sh*t hits the fan, the business enter in the "ain't nobody got time for that" mode and they urge for the fix.
For example, our software is sold to enterprise customers. These customers use the software to run real-time reverse auction procurements where the total value of the purchase can be anywhere from $250,000 to $20,000,000.
The results of the reverse auction events are awarded in the form of a contract between buyer and seller. If a software bug causes an incorrect calculation, it's a big problem (0.1% error on a $10M purchase is a $10,000 error). And by big problem, I mean that weeks (maybe months) worth of effort from our team, the customer's team, and several vendors go down the drain.
Put simply, a bug in our bid core could cost us $40,000-$50,000, assuming the lost business assessment is limited to the single failed procurement event. Looking at the total value of a lost customer, you're easily talking $250k.
Because of this, we have a very long QA cycle. Outside of automated tests, our software is touched by humans (a lot) before it goes to production, and production releases are less frequent (once every couple of months).
The frequency with which you deploy, and the amount of effort you put in to QA is determined by the cost of failure. We're in a rather unenviable position of being a small fish in the enterprise market, so our cost of failure is huge (we have a small number of huge clients). Thus, we invest significant effort in QA and slow-roll to production. Your situation may be entirely different.
On the staging server, I observe if everything works well by going through some use case scenarios manually. Once I'm convinced that everything works well, I deploy the new version to the production server for my beta testers to try out and give feedback.
So cycles of "new versions on production" can currently take anything between a few minutes (really quick bugfix, "OMG the website is not working at all!") and a few days (new feature)
One could complain that there should be a test suite running unit tests and integration tests, taking care of making sure everything works well. But I have two observations that make it impossible for me to rely on a big automatic test suite:
* as my website is still technically in an explorative mode, constructing test cases is wasteful: my website's architecture changes quickly to adapt to new insights. I rather spend my resources on developing features than on a test suite that doesn't work with the new architecture. * Subtle differences on my production server (limited server resources, different HTTP request behavior) make it impossible to rely on a simulated environment for my integration tests. Therefore, I have a staging server with the exact configuration as my production server. The only difference are IP address and domain name. These differences shouldn't break anything in my website, but you never know until it breaks the first time :)
Work - Continuously, can be within minutes:
* Team commits to SVN trunk, tests run + packages built. Manual tests run (can take some time and more fixing). Merge to production branch, tests run + packages built then pushed out to servers.
Personal - Depends on time but few times a month on average:
* I work in Python and do my own DevOps stuff. (Not a Heroku/Appfog/etc fan..) I use SaltStack (saltstack.org) to keep a 'template' that I can apply to a blank Linux Server (AWS|Digital Ocean|Local VirtualBox for dev etc) and it will set it up for me, same layout, every time. Kind of like Puppet/Chef, but in Python. (It's awesome - definitely check it out)
* Commit to BitBucket (for private repos) -> Jenkins pulls code down -> Runs tests, displays coverage etc -> Builds RPMs & Debs -> Salt pushes out to any connected minion (yup - it also has arbitrary command execution).
I'm going to write a blog post on the entire personal setup if anyone is interested.
Also checking out Linux containers (specifically docker.io) to see if I can speed this up.
But, I used to work for departments of the UK and US governements and it wasn't unusual to have such locked-down and managed environments that it would take 6 months to release some code. It was also expensive (the sponsor department would have to pay for validation and testing by a third party).
The consequence of this is that we really only deployed every 1-2 years.
When you've only got a few, experienced developers, it's much easier to ask "hey, how do you guys feel about the changes?" and push if everyone's fine. With larger teams or less experienced people, you need to go through more testing, which slows things down.
The end result, is that to have a patch deployed you'd have to get the patch, the whole codebase, and everything replicated and proven elsewhere (with realistica data and use cases), and a third party was contracted to do this.
It became insanely expensive to do even the smallest thing to make users or project sponsors happy... we couldn't give them code in any timely way, so everything became workarounds.
And you'd always find some project that would fight with this so much that they'd start writing in back doors. Auto-generating a WSDL on one occasion revealed a method for running arbitrary SQL. I guess some dev got sick of waiting 6 months for the next answer they needed.
I was more referring to how easy it is to push frequently to production when you have a small team of good developers, though. As the team gets larger, the ease of pushing to production drops (leading to your example above, in the long run and with crappy dev culture, I guess).
Live editing and debugging on the servers is strongly discouraged. But still have to do echo print_r from time to time.
Lesson learned trough the years of practice - there will always be big enough lump of fecal matter that when it hits the fan it will make you abandon all of the best practices for a while.
"support fixes" - ASA(F)P, after 2 rounds of testing and several sign-offs + Management approvals.
"Change Requests" - Deployed after an impact+risk-assessed notice period (between 1 - 14 days), and after the above testing/sign-offs/approvals
"New Release" - Should be Quarterly, after several rounds of testing, sign-offs, agreements from all affected business units... but there's been a big push on the app recently, so there's been 3 releases this year so far.
And then we have to go through all the testing/sign-off/deployment stages again with any Joint-Venture companies that have their own installations due to local data laws.
However, with all these people talking about deployment, I just wanna hijack (sorry!) and ask if anybody can help with my current deployment setup:
- Branch into a new feature (refactor-javascript for example) - Commit constantly - Rebase with master then merge into master - Push to remote repo - SSH to server, and then pull from the same repo - (.git isn't exposed via HTTP)
What's the better way of doing this? If you could help that'd be great :-) Drop me an email at andy@fine.io
I have a server setup with a headless git repo and non-headless one. When I push to the headless repo, the post-receive hook goes to the other repo, pulls from the headless, runs tests and restarts the server
So to deploy I just do
git push deploy master
I always found that the hook never really worked for some bizarre reason. Even something as simple as:
cd /home/deployer/wordpress/wp-content/themes/my-theme && git pull origin master
failed.
My hook is:
cd $REPO_DIR
unset GIT_DIR
unset GIT_WORK_TREE
git checkout deploy && \
git stash && \
git pullWe also never SSH into a server, and instead use Capistrano to handle this for us. Capistrano works great with Rails, but even other frameworks have plugins to handle this. And if you're new to capistrano, take a look at http://capo.io (shameless plug: we built this) for readily available recipes for all kinds of tasks. We use Capo for all our projects, ranging from static sites to jekyll to sinatra, rack and rails apps.
I wrote a quick fabfile yesterday that automatically commits/merges my local changes, pushes to a private GitHub repo, then SSHs into my production server, pulls down the latest code and restarts the web server. Gives me feedback along the way too.
Don't use fabric for deployment (even though it's a fantastic tool for many use cases).
The only changes I'd make would be to wrap it up in a script so you have one-click deploys, and possibly implement the 'copy and symlink' strategy that Capistrano uses so you have minimal downtime during deploys and instant roll-backs if necessary.
I only hand over work to him on the proviso that my work is ready "pending end user acceptance testing". Of course there's still the traditional 2nd of january deploy untested code manouver they seem to love. Last time they pulled that it cost them at least $40k but they were warned and we weren't fired.
If you're not using Jenkins you don't know what you're missing, but I can tell you it's warm and sunny :)
here, we deploy a major update to the system every 6 weeks. All database changes go into the major update.
We release patches to production frequently. In first week of the major update we might do 5 - 6 patches per day.
This is not easy for us, as we do not have everything on the web.
We have a java application that is patched and downloaded from our users so we want to avoid forcing them to download the new stuff too often.
if it was a web application the impact of the patch would be minimal.
Though it consumes a lot of bandwidth, but automation is the key here. It can be minimized if you have a CI tool that can build the product and deploy on the machine itself.
Major changes: every 5 weeks (build for 4 weeks; test for 1) Minor changes: every week (urgent and emergency changes) Bug fixes: as needed. usually 0x/week but can be multiple times a day if necessary.
Mobile app releases: rarely. The belief is that frequently updated apps are considered a bad thing.
Regarding your backend, are you using any type of CI?
new version around once a month to production, datafixes whenever they are urgent. BI/DW & reporting side with quite a bit faster cycle, around once a week, as the requirements of the extremely complex metrics and dimensional data hierarchies are still evolving.
Personally anywhere from a few times an hour to a day or so. I try to keep all development tasks small enough to deploy on the hour if possible.