The issue, obviously, is the app store submission/approval cycle which makes changes slow to reach users. (even without that, you probably wouldn't want to make users update their apps every day).
If the app is mainly web-based, it's pretty obvious, but what about native apps?
We had an app launch on launchly awhile back that was basically trying to be the iPhone version of GetSatisfaction or UserVoice. You could plug in their code into your app and it would handle feedback and bug reports for you.
Maybe there's a market for something similar for A/B testing on iPhones. If it's a tricky problem to solve, even better cause then people will pay you for it.
Initial thought w/out knowing anything about the security model is can you set up config files or even remote code on a server somewhere and then access it from your code on the iPhone using an API key? If you can then you could maybe set up multiple config files on the server and have your code load different versions randomly and report back statistics...
Of course that brings up an interesting point. What would you be A/B testing in an iPhone app? The user has to have already downloaded/installed your app. At that point there's really not any landing page or funnel to test like there is on a website. Obviously you can still A/B test internals just to improve the app which will affect ratings which can affect downloads (conversions).
Anyways, just thinking out loud here.
Note:
i) Reasoning behind not having it as first screen as it really slows the application launch.
ii) You should make sure that if webservice call takes too long, just pick one default which is shown to user instead of waiting for result
iii) if you really want to do it for first screen, load and save the result in user settings for next launch while the application is running
1. Prepare two versions of each feature you want to test
2. Generate random integer on first launch
3. Save to NSUserDefaults
4. Present A/B based on the value of a particular bit in the random integer
5. Collect statistics and send back anonymously via NSURLConnection
Question is probably better suited to StackOverflow
- Do fake A/B testing for your AppStore product page, toggle once a week and compare sales.
- For the product itself, try usability testing face-to-face with potential users, using ad-hoc distribution.