I was talking about continuous deployment with some colleagues earlier and we were wondering whether we'd be able to deploy a version of the app to only a subset of our users on app engine.

We didn't think it would be possible, but it looks like this Traffic Splitting feature could be (ab)used to make it happen after all.