Always ship trunk: Managing change in complex websites | Hacker News Reader

Always ship trunk: Managing change in complex websites

paulhammond.org

138 pointsbrown9-223 comments

For what it's worth, this is pretty much exactly what we did at the last web property I ran with ~500 million monthly pageviews. We had a configuration framework that allowed us to flip bits on the fly (and, importantly, flip them back), without a deploy, in ~1 second across all of our app servers. The app servers listened to a message queue which would broadcast changes of configuration values.

This one thing alone made a huge difference in our engineering velocity.

If you're looking for a pre-built component to do these things for you today, check out Apache Zookeeper: http://hadoop.apache.org/zookeeper/

Yes. If you're in this situation Zookeeper will change your life.

Good points, and we're doing almost exactly that, with one difference.

I think it's worth having 2 branches you ship from: master and production (or, in reverse, master and development).

The reason is that this way, if you're in the middle of some relatively large and untested changes in the development branch, you still have a copy of exactly what's on production to work on in case there's an urgent patch that needs to go out.

This is not strictly necessary, of course, but we've found that it's saved us some hassles.

At the end, he asks "What would a version control system for a web app look like?" Perhaps that's only a rhetorical question, but I think he's already touched on a partial answer: it's a configuration management system. A good configuration management system will help you keep track of what's currently deployed where (although variation per login id is generally out of scope) and help you be sure that the requirements for deployment are met. The source for the configuration is normally kept under revision control for non-branch elements that revision control does well: why the change went in, who put the change it (in case the "why" was incomplete ;-}), what the change was (centralized diff), and when the change went in. When can be important for those little used but essential features that you find out stopped working some time ago, and you want to find out what made it break.

He advises discipline in removing unused configurations. I'd like to hear what sort of dependency analysis tools support that activity.

Doesn't Google App Engine provide some of this for free? You upload versions, change which one is deployed, and can easily roll back at a moment's notice. It's not as swift or sexy as multiple configurations in the same code, but still far nicer than managing it yourself.

This will seem a lot less heretical if you do a lot of A/B tests, since if you're already supporting 2^13 versions of the code running simultaneously, heck, what is a couple more.

It's easy to miss, but this talk is given by a Flickr engineer and is how they deploy their code. Pretty big recommendation.

Former eng manager actually.

In this preso he's focusing in on just one aspect of how Flickr does things. He did a more wide-ranging talk with ops guru John Allspaw last year.

http://velocityconference.blip.tv/file/2284377

For instance, as people noted already, he's advocating testing features on live servers -- in a controlled way. This doesn't work unless you have the kind of dev/ops cooperation that he describes in the above video, as well as practices for quickly rolling back changes and a culture of not blaming people.

Wow, I just learned a new idea. Test SaaS on production servers. Sounds scary but makes sense. Fast driving is not always dangerous driving. Great post.

Sounds like a bit of tool support would be useful.

Note that the UNIX command patch has a -D <symbol> option, which results in the patch only being applied if the (C/C++) program is compiled with -D <symbol>, so this idea is not actually all that new...

After much experimentation over the years, I've found the best approach as follows (using git):

- always ship master

- always develop using topic branches (branches dedicated to a single feature)

- deploy topic branches to dev/staging (with large visible footer label text)

- don't leave old topic branches around on your central git server (delete as soon as its tested and merged back into master)

Good 5 minute read.

Last slide asks a great question:

What would a revision control system built for supporting deployed web applications be like?

My first thought is that this should be managed using a Factory pattern. If one is truly doing A/B testing, then this is manageable for N = 2, and a lot of repeated conditional code with the same if-then-else statement will disappear.

If they are A/B testing a lot of things, then this can get unmanageable. For that, how about a collection of functions? (methods? I am not trying to be language-specific at all here.) Have a facade which stores references to the actual functions. This is much easier if your application is structured in a functional or nearly-functional way.

Writing 100 if-then-else statements without making a single error is doable. But making the conditional logic disappear and just having to manage the methods themselves and a configuration is much easier.

That was my thought at first as well, but probably for illustrative purposes, it was easier to do if-else statements. The more structure there is, the higher the learning curve. For a presentation, it probably wasn't necessary.

In practice, it turns out to be easier to do with if-else statements as well.

A factory pattern is useful when you know ahead of time along what dimensions your system will change. You can build an interface with methods for each of those change points, then call out to those methods when appropriate. For each experimental variant, you provide a different implementation and instantiate it based on experiment selection at the factory.

Unfortunately, these types of configuration changes, by definition, tend to be experimental. Meaning you have no idea how the app is going to change to accommodate them, you just have an idea that might or might not help users and you want to test that hypothesis as quickly as possible. Rearchitecting your app to build the appropriate factory interfaces takes significantly longer than just throwing in an if, and it's all wasted effort if the experiment comes back with a null result (as many do).

If you find yourself making experimental changes to the same areas often, then you might want to look at introducing a Factory. But design patterns in general are something you reach for once the system has matured, not something you do when you're still experimenting and exploring the domain.

If you read my post carefully, I propose a Factory, then reject it in favor of something simpler.

My proposal reduces the refactoring to "this block of code becomes a function call." There's still a piece of conditional logic, for each dimension, but it only gets written once. The application code gets smaller by applying this, except for one startup routine where we set up the thunk. Overall, it's actually less code than sprinkling conditionals everywhere. You could set things up so you change the configuration of all your A/B experiments in a single file. It's not so much a pattern as it is a reorganization of the logic along a straightforward application of DRY.

One might still want to stick with the conditionals everywhere if most of the code needs local state. This is probably an indication of a need to refactor, though.

Thank you - very concise and will help me personally in a current initiative to drive some points home!

I suppose some details would be appropriate....

I'm dealing with a suite of what used to be plain-old websites that slowly, over time, started to gain functionality. PHP crept in there (just for includes) - the web team went from one person to an entire department in a short period of time, and some functionality keeps creeping in there - some of it done by developers, some of it done by web PHP guys who shouldn't really be writing code. Incremental direct updates to the site on a daily basis are a regular thing by far too many people.

I'm sure many people have the same story.

The challenge is in taking the current situation and turning it into something manageable with release control, testing, and so on - because it's at a point where a simple mistake by a designer can (and has) effectively taken the site down, and due to the nature of the update process, it's hard to find out 'wtf' just happened. The GOOD part is everyone is on-board with changing things (no politics).

The hard part is figuring out what those changes are - as soon as we started trying to fit this into the standard developer workflow, it was obvious that we were adding far, far too much un-needed technical bureaucracy (do we have a word for that?) for no real benefit - and the idea that the release and deployment tools are where the real benefit is is really becoming apparent - the version-control system is necessary, but it's not strictly the most important piece.

So for me, these slides (wish I could see the talk that went with it) really helps cement that I'm not insane for thinking differently about how this should work.

btw: scribd works really well now.

92 slides is a bit long for the points the author is making, although it's hard to say without hearing the talk.

Basically the author prefers GIT and mercurial over subversion because of easier branching, and argues for having multiple versions of the backend running in order to switch between them with with configs in your code (eg. cfg["use_new_awesome_backend"] = 1) in multiple variations (eg. if (rand(0,99)<10) cfg[".."]=1, etc).

>Basically the author prefers GIT and mercurial over subversion because of easier branching

That wasn't exactly the conclusion I got out of it. I felt like he expressed a slight preference for git and mercurial, but at the end he asked for people to envision what a web/SAAS version control system should really look like, since none of the 3 fit that bill, in his opinion.

(I agree with the second half of what you wrote that I'm not quoting here.)

True, I missed the point in the first part. Thanks for pointing that out.

Most of the slides are blank, single sentences, or changes in highlighting. I'm sure they were flipping through them pretty quickly and just using slides to give certain phrases extra punch.

See also: Lawrence Lessig's presentation on Free Culture (http://randomfoo.net/oscon/2002/lessig/free.html) That's probably 1000 slides, but they are used expertly to deliver a clear, concise message.

Reply on news.ycombinator.com