> 3) How do you determine which devices and OS versions we should test on?
> This should generally be an easy question for the candidate. Good candidates will point to app analytics as the best measure, looking for the most used devices for their particular app. Another good answer would be to check the app reviews where people might have complained about specific issues happening on their devices. Less creative candidates might simply suggest “looking at the top devices” on the market
Looking at top devices is also incredibly important, and it seems foolish to dismiss that strategy.
What if your website doesn't work on iPhones, and so you have no iPhone users? What if, because people can't even sign up on an iPhone, they don't bother complaining about specific features not working?
Surely it makes more sense to test based on your target market, rather than the subset of people who are self-selecting to use your product?
Ideally you should be looking at both usage and market data. If your target market contains lots of iPhone users, but everyone's on Android, this points to a potential problem.