maximedupre

I'd be curious to know what the other a/b testing tools are doing regarding this problem if anything.

Hey Joseph, thanks for the comment. To be honest, this scenario never crossed my mind once xD. I have not implemented any solution to this problem... and as I'm writing this comment I'm not sure what the solution would look like. Surely the other big A/B testing tools have solved this and I could inspire myself from their solutions. Splitzy is very new (a couple of weeks old), so I haven't really covered all the possible use cases. It's very simple right now =).

:D

It's better if all your experiments are independent from each other however.

Interesting project Falak! Keep it up

This looks like a handy tool :D

Thank you so much dude :D Onwards and upwards!

Thanks Daniel, I guess xD That was quite cryptic, but I think I get it haha. Cheers!

That's an interesting app :) What motivated you to build peerdiem?

You understand Birdy's mechanism correctly :D

> How do you make sure the same user who was on A variant yesterday doesn't get the B variant today?

You can't. You make the point that the stats generated by Birdy are not 100% accurate, but they're the best we've got given the limitations of the Twitter API :D

I make the point that the data is good enough, and statistically significant enough (especially if you let the test run enough) to help you find the better profile version.

I agree that this subset of visitors that may come twice or more can skew the data.

Birdy's stats are not to be taken as the absolute truth, but as solid clues as to which profile version it is alternating between is providing the best results.

I'm looking forward to improving Birdy as Twitter adds more capabilities to their API!

I suppose that the issue is that a single user could see both versions, which could skew the data.

I'm not sure I totally understand why, because if a user follows you after seeing the other profile version, it might be because he preferred this other version.

But conceptually it makes sense to eliminate as many variables as you can to isolate the components of the test.

Thanks for the comment :D

> There is no “testing” going on here, unfortunately. You’re just taking turns placing users in the treatment or control group, arbitrarily, depending on when they visit the page. How can you measure any treatment effect when everyone is part of either group?

This is not a perfect solution, but I have found that it is good enough to be able to identify clear winners (if there is actually a winning version). A lot of followers won't visit your profile multiple times anyway. They visit it, and they either follow it or they don't - and they will of course be influenced by the currently displayed version :). They won't just come back to your profile over and over again for no reason, but if they do, they will also convert better on the version they prefer. So a version might nudge the user into following you while another one might not. I would disagree that this is not testing. I think it is for the majority of the profile clicks you receive.

> I guess you could try something like switchback testing, but I’m not convinced visits to the average Twitter profile will yield enough samples.

I don't think that would be possible with the current Twitter API capabilities anyway.

> I think it’s a well-executed idea, but I don’t think it’s fair to sell results under the guise of statistical validity when they don’t appear to have that. (although it’s just Twitter profiles, not eg medical treatment, so no real harm done)

While the results might not be perfectly accurate, I think they are accurate enough to provide value, especially if you let the test run long enough to get a big sample size. I personally use Birdy (obviously :D) and I have noticed much better conversion, which is why I'm confident.

I'm looking forward to seeing new capabilities appear on the Twitter API to always make the process more accurate though.

It doesn't seem to be available for the Mac app store however :/

Thank you!! I wouldn't call this a launch though, I was expecting to post into the void xD

Thank you!!

I'm not the one that will create this, but I agree. The more you can A/B test everything, the better xD

I feel like that's less of a problem because the user is more likely to convert on the version that he likes more, which would still provide accurate data.

But yes, there is no perfect solution with the limitations of the API :D

Thank you! Yes, custom built with Tailwind. First I wanted to use a landing page template, but then I decided it would be easier to just build it myself hehe

Definitely double down on the one you already have. Starting from 0 is way more difficult than starting out with 300 followers.

Transform your profile and make it super interesting (work on your banner and bio, then have a good pinned tweet once you get a somewhat viral tweet). Then engages daily with other accounts - that's where new people will discover you.

Oh right, that's actually a better technique! Thanks for the idea.

Thank you!

> I think the only downside to this approach is that there might be some effect based on time of day or the interval affecting your results.

That is definitely true. I'm about to start alternating the versions every 5m to mitigate this. The closer I can get to 0m, the more accurate the results are. This way even if you get a followers spike (let's say you get a viral tweet), the followers will be properly distributed between each version.

Start building a strong Twitter presence! Also engage with the IndieHacker forum.

Oh! That's a good idea. I didn't think about that.

I think at some point I'll have a free plan for everyone (with limited features). I just need to finish implementing all the profile components and then enable only some of them for the free plan.

Thanks. The secret is that you need to tweet a lot [of good stuff] :D

Thank you!!

I wasn't expecting this post to reach the front page of HN, I usually get ignored on here xD

Thanks, did quite a bit of work on the landing page :D

I got a bunch of great feedback on Twitter as well, so I could polish it a bit.

That is correct - it switched your profile version at a regular interval.

Indeed, that's the only way I could do it. The Twitter API has its limitations :D

I didn't know there was a specific definition of A/B testing. I'll see if I get more complaints about the terms I use ^^.

To me, that's still A/B testing - that is I'm testing a version A and a version B and then report on which one does better. I guess the way I'm doing it is different :D

I have about 15 users. Because they want to optimize their profile and perfect the funnel that converts profile visitors into followers :D Or test out which profile converts to more clicks on their bio link.

That is the plan Mike! And I'd rather have fun while building it :D

Thank you sir. Try it out sometime

Thanks :) Now let's see if people want to actually use it :D

The only reason I built Birdy is because I'm totally a fan of the platform :D

I'm not so much a fan of the App Store platform.

As far as Twitter Profile A/B Testing goes, it is in my plans to add AI-based suggestions (based on the top performing profile versions) :)