Not that it matters - this solution is probably the right way to go without building something into Twitter itself - but the more data/stats oriented folks may be confused or irked by calling this “A/B testing”
Indeed, that's the only way I could do it. The Twitter API has its limitations :D
I didn't know there was a specific definition of A/B testing. I'll see if I get more complaints about the terms I use ^^.
To me, that's still A/B testing - that is I'm testing a version A and a version B and then report on which one does better. I guess the way I'm doing it is different :D
> I think the only downside to this approach is that there might be some effect based on time of day or the interval affecting your results.
That is definitely true. I'm about to start alternating the versions every 5m to mitigate this. The closer I can get to 0m, the more accurate the results are. This way even if you get a followers spike (let's say you get a viral tweet), the followers will be properly distributed between each version.
But yes, there is no perfect solution with the limitations of the API :D
You gotta be sure of what you know. Be sure and be wrong, better than always being unsure.
I'm not sure I totally understand why, because if a user follows you after seeing the other profile version, it might be because he preferred this other version.
But conceptually it makes sense to eliminate as many variables as you can to isolate the components of the test.