This approach makes a lot of sense. I tried something similar with an iOS app and was surprised to find that Apple frowns upon this practice. They'd prefer that developers do usability testing beforehand instead of native A/B testing in the wild. Does anyone know if Apple has clarified this position?
It would be much more developer friendly to encourage this and would ultimately lead to better experiences for users.