Staging Servers, Source Control & Deploy Workflows, And Other Stuff
kalzumeus.com
kalzumeus.com
For anyone not already aware of it, I recommend checking out Git Flow. It's a set of git extensions that standardize the git workflow:
https://github.com/nvie/gitflow
Some articles:
A developer that worked for me did exactly this a few years ago, only instead of ringing numbers he sent them overcharged SMS messages. I had to call up every single affected customer and explain to them why they had just received 50 SMS messages that cost them $5 a pop. After that I of course refunded the money - only problem was that the SMS gateway charges 40% on each transaction, which I couldn't get back.
Very expensive mistake.
Given that each environment has an app server, a web server, two or more batch servers and a database, I can vouch that a fubarred deployment workflow is hell. You end up with no baseline to tell if your process is bombing due to configuration or code.
Also, not to pimp my own stuff, but I wrote a thing about generating test data with Ruby some time ago. I've used this strategy a number of times and it works really well: http://larrywright.me/blog/articles/215-generating-realistic...
It is so useful to have very similar setups in staging and production.
In particular, I really try to avoid having a different architecture (eg: 32bits vs 64bits, or different versions of ubuntu, or passenger just in production etc). It makes it easier to catch issues earlier.
It is virtually impossible to understate how much using source control improves software development.
Shouldn't that be "overstate"?
I would say "from bare metal" rather than "from the bare metal"
Not saying that's a bad thing... I just think the distinction is very important. :-)
One-click rollbacks. It's really, really important that when you deploy a release to the production servers, you can un-deploy it with a single click or command. That means all changes should be logged, and all the old files should be kept around until the next release. You hopefully won't have to use this often, but when you do, it's a lifesaver to be able to say "Okay, we'll rollback and fix the problem at our leisure" rather than frantically trying to get the servers back online.
Staging/production configs. If you do need to have differences between staging & production configs, try to limit them to a single overrides file. This should not contain app config that changes frequently, and should be limited to things like debug options and Patrick's "don't e-mail all these people" flag. Check in both the staging and production config overrides, but don't check in the actual filename under which the system looks for them. On the actual machines, cp the appropriate config to the appropriate location, and then leave it there. This way it doesn't get blown away when you redeploy, and you don't need to manual work to update it on deployment. (I suppose you could have your deployment scripts take a staging or production arg and copy it over appropriately, but this is the poor-man's version.)
Deployment schedule. I'd really recommend having a set, periodic deployment schedule, maybe even run off a cronjob. The problem with manual deployments is they usually happen only when people get around to it, and by then, dozens of changes have gone in. If something goes wrong, it's hard to isolate the actual problem. Also, deploying infrequently is bad for your users: it means they have to wait longer for updates, and they don't get the feeling that they're visiting a living, dynamic, frequently-updated website.
The holy grail for deployment is push-on-green. This is a continuous-integration model where you have a daemon process that continually checks out the latest source code, runs all the unit tests, deploys it to the staging server, runs all the functional & integration tests, and if everything passes, pushes the software straight to the production servers. Obviously, you need very good automatic test coverage for this to work, because the decision on whether to push is completely automatic and is based on whether the tests pass. But it has big benefits for both reliability and morale as team size grows, and big benefits for users as they get the latest features quickly and you can measure the impact of what you're doing immediately. I believe FaceBook uses this system, and I know of one team inside Google that has the technical capability to do this, although in practice they still have some manual oversight.
Third-party software. I know Patrick recommended using apt-get, but I'm going to counter-recommend pulling any third-party code you use into your own source tree and building it with your own build tools. (Oftentimes you'll see all third-party software in its own directory, which makes it easier to audit for license compliance.) You should integrate in a new version when you have a big block of spare time, because it'll most likely be a long, painful process.
There are two main reasons for this. 1) is versioning. When you apt-get a package, you get the most recent version packaged version. This is not always the most recent version, nor is it always compatible with previous versions. You do not want to be tracking down a subtle version incompatibility when you're setting up a new server or deploying a new version to the production servers - or worse, when you rollback a change. (If you do insist on using apt-get, make sure that you specify the version for the package to avoid this.)
2.) is platforms. If you always use Debian-based systems, apt-get works great. But what if one of your devs wants to use a MacBook? What if you switch hosts and your new host uses a RedHat-based system? The build-from-source installers usually have mechanisms to account for different platforms; open-source software usually wants the widest possible audience of developers. The pre-packaged versions, not so much. And there're often subtle differences between the packaged versions and the source - I recall that PIL had a different import path when it was built & installed from source vs. when it was installed through apt-get.
Counter-counter-recommended. This is needlessly duplicating immense amount of work that distro packagers do.
> You do not want to be tracking down a subtle version incompatibility when you're setting up a new server or deploying a new version to the production servers - or worse, when you rollback a change.
This is why LTS releases exist. If you're locked to Ubuntu 10.04, then you'll be using the packages that come with it until you're ready to make the significant leap to the next LTS version three years later.
> If you always use Debian-based systems, apt-get works great. But what if one of your devs wants to use a MacBook?
Then they can suck it up and learn how virtualbox works. Even versions you've hand-chosen are going to exhibit cross-platform differences that will make them fail to reflect the reality of production: case-insensitivity and GNU/BSD differences are two such things that come to mind. (Indeed, both of these have been encountered in the last few months by one of last few the VM-holdouts at work.)
Added advantage of using virtualisation is I can easily trash and rebuild my dev environment whenever I need to.
> Counter-counter-recommended. This is needlessly duplicating immense amount of work that distro packagers do.
I think that distro packagers do far too much work: they sometimes do not include compilation options that are very useful, apply distro-specific patches and add too many dependencies. And when you have a problem, they should become the primary point of contact, not the "upstream" writer of the software which has zero control on how it is packaged.
For the vast majority of people the distro packages are fine, but for some people the distro packages are an inconvenience.
Patrick's suggestion to keep a log with all the setup steps for the production was already a life saver. Two days ago I remembered that my server was running 32bit code and I was going to run MongoDB on it. Whoops. Complete reinstall in 48min, worry-free.
I'm keen on git, and plan to use it to deploy to production. I actually rsync to test/staging server doing development[0] (to avoid having to commit knowingly broken code just to be able to deploy on testing/staging server), but I use git to manage the code, and to deploy it. I have a clone repo on the production server with a production branch, which has a few commits on top of master, in which I committed the production-specific configuration.
Deploying on production does roughly: 1. check out the latest code from master 2. rebase production on top of master in a temporary repo to catch any rebase problems (because I don't want to merge master into production) 3. run unit tests in a temporary repo (sadly my tests only test the backend, not the web ui; I plan to improve in 2011 :) 4. rebase real production repo 5. make a new tag (for almost-one-click rollbacks) 6. restart whatever services need to be restarted
This is automated by a simple shell script which aborts at the first sign of trouble.
Regarding the 3rd party packages versioning: I use system packages wherever available. I don't have automatic updates though, and I don't use a system having rolling updates (I'm on Ubuntu Maverick). I had to manually rebuild two packages: nginx (to include upload module) and Tornado web server (the one in Maverick is too old for my purposes). This was pretty straightforward, and I've recorded the exact steps in my server setup log.
[0] my app involves callbacks from external services, so I can't test it on my laptop; my development workflow is "save in editor, rsync, see whether it works", with services in debug mode reloading themselves as soon as change is detected.
Do you protect the dot-files from being accessed via the web? I was at a security conference recently and a speaker mentioned a number of companies had source code accessible over the web because they served directly from a repo and didn't block accessing the VCS files.
So that person logs in, sees a list of green things, and clicks on all of them. Like marking messages read in gmail.
Never heard of this before, it sounds cool.
The main purpose of anonymisation, in this case, is to make sure you don't send testing emails to clients. So actually, the only kind of scrubbing you really need to do is to make sure every email/phone number/twitter handle/outwardly communicating piece of data is replaced by a test email/etc.
The hardcore anonymisation that banks use is only necessary because there is an actual security and reputation risk if the data is leaked by some random developer in India (or some angry developer in London). In the case of swiss banks, they are also legally obliged to scrub that data when using it internally in locations outside of Switzerland.
However, for the purpose of a startup with 1-30 ppl, most of whom have access to production anyway, there is no sense in doing that kind of anonymisation. The only risk you're protecting yourself against is sending hundreds of testing emails to your customers.
The solution there is to have robust access control to all of your servers.
Incidentally, The nice thing about having the infrastructure to deploy a replica of your production environment is that it's probably not much harder to deploy multiple scaled-down versions cheaply, so that you can do two stages of QA. You can do all possible testing in an environment with a fake database, then for the real staging test use the scrubbed production version.
This will also be useful for developers willing to test changes with the real data volume.
EDIT: after reading patio11 comment, I can only emphasize the importance of using a "white list" approach here, ie carefully picking the fields you will keep, and adding ETL screens. You have been warned :)
It may be that I've been working with government data too much, but people's dev environments are generally far less secure than their prod environments, and I personally don't want to be the guy who declared that the scrubbed data contains nothing sensitive and be wrong, whether or not that data gets out into the wild.
That's something worth of having as well.
So now we finally know patio11's grand scheme!
But seriously, thanks for the writeup. I am using the lazy man's version control (Dropbox... ;-) ), but I definitely need to more to Git ASAP. I guess before now the time spent learning and setting Git up was better spent doing something else (at least in my mind).
I do agree regarding how nice it is to use Heroku. My only issue with them is that they only support Ruby and Node.js, so I need to take my Clojure applications elsewhere.
Staging environment must be IDENTICAL to production or ridiculous and disastrous errors will occur when you least expect them.
Source control just provides lots of benefits for logging and rolling back and around. Git and Mercurial, as distributed version control systems, just happen to be easiest since you don't need to really setup a repository, and just have all the data on local disk. It really is practically free.
Go to your source directory now:
1. git init # makes the directory a git repository
2. git add . # adds everything in that directory
3. git commit # commits it
If you don't mind going for mercurial instead of git then take a look at hginit.com. Then if you're on windows, go and install tortoiseHg.
https://github.com/apinstein/git-deployment/
Seems pretty nifty.
If you're working on your own then commiting frequently is fine. If you are working with a team then that's a lot of information for your colleagues to process, so it's a good idea to condense your changes down a bit when you commit to the main branch. I think that's what the article is getting at when it talks about feature branches.
With git the more important questions: When do you push?
Not as often as possible. I repeatedly regret a push, because i need to commit a revert/change commit afterwards. If i had waited, this could have been cleaned up before pushing.