Show HN – Apptimize: A/B testing for native Android, iOS apps | Hacker News Reader

Show HN – Apptimize: A/B testing for native Android, iOS apps

newton.apptimize.com

126 pointsjorlow50 comments

For folks interested in mobile A/B testing, also see an open-source alternative by Clutch: https://github.com/clutchio

For what it's worth, using Clutch was one of the reasons we ended up making Apptimize. We ran into several bugs including one that was crashing the app we were working on and issues with its probability distribution. And the reporting was very simplistic.

Are you using Clutch now? If so, it'd be awesome if you could give us a try.

Additionally, incumbent mobile A/B testing service http://www.leanplum.com -- I've used them for a while now and it's a pretty robust system with good support. Worth checking out IMO.

ClutchIO is not native, which is why something like this is needed.

This looks interesting - although having never really considered using A/B testing on native apps before I did spend a few minutes looking around the site for examples of things to track. Maybe you could show an example project on the site showing what was changed and how that affected sales, etc.?

Part of apptimize team here. We want to do that! We will add more to our site; currently it looks deceptively slick bc our designer is awesome but it's really a first iteration.

Right now if you sign in, you can see an example experiment. How does it look to you?

Yes the example gives a much better idea of what's going on. Although I did have to create an account to look at it...

Would be nice to have that accessible from the main page.

I couldn't find any pricing information even after registering. It only says its free to try, but no further clarification in the docs or the FAQ.

While it looks promising, I can't go and integrate this into my app without knowing if, when, and how much you're going to be charging for it.

We haven't announced our pricing yet because we're constantly adding new features our users request. Email me at nancy at apptimize dot com and we'll give you a special deal we've been giving to early adopters.

The lack of information on pricing is an instant back-button-trigger for me. I have no idea if you (eventually) plan to charge $5 per month or $200 per month. If it's in the lower end, then it can be worth evaluating. If it's a few or several hundreds per month, then I have to pass until I'm earning enough to afford it.

Do people actually do A/B testing on iOS apps? I feel like the two-week development cycle really hinders your ability to conduct valuable A/B tests with your users in time to actually make any change.

Some of the more forward thinking folks (Socialcam and HauteLook for example) have rolled in house versions of these, but Apptimize is one of the first end to end solutions I've seen. But A/B testing in mobile is definitely something a lot of folks are thinking about.

We faced the same problem, and built an A/B Testing Framework at a Hackathon this month that allows you to just drop in a line of code and run all the experiments through a web interface without app updates.

The framework is not quite ready to ship yet, but we built a fully working proof of concept, and are looking to complete it and put it out there.

You can read more and watch a video demo at the website: http://fliptest.io

Take a look at http://useartisan.com/

Basically, it removes the 2-week development cycle, and you can essentially make changes in real time. I've seen it demo'ed, and it's much more impressive than the website makes it out to be.

It sounds interesting, but how do they get around the app store approval process?

You compile once, and change the app as needed. You don't need to resubmit.

You're right the app store is a barrier but people still do it. That's one of the advantages of our product, that you can start and stop experiments so you can put in as many variations as you want. Currently people just suffer through the really slow time frame.

We build in DB-driven tests into our app so we can make changes at will once we find a winner. Then we just clean up the code to remove the test in the next iteration.

I would like to see some code examples.

If you register (takes 2 seconds) there's an example project where you can see the integration steps with the exact code you need to paste in. You're right though, we should definitely add examples to the main site.

I'd recommend not forcing people to register to see generic information like this - I know HN is biased towards "Connect with Facebook to login", I would assume most are also biased towards registering to see content - I know I am.

Ok you guys are right. We're adding this to our site right now.

Updated: https://newton.apptimize.com/faqs.html

It'd be nice if the text were selectable instead of an image.

It also looks really bad on a retina screen

Thanks :)

Our organization often comes up with its own secondary metrics based on user data (not just CTR or usage). Is this possible when using Apptimize?

Part of apptimize team here! We let you track whatever goals you want, so people measure stuff like "user returned within a week" and "user used app." What goals do you want to track?

That really depends on what we find to be important metrics. Flexibility is important since we need to be flexible too.

Interesting, Artisan mobile is also in this space http://useartisan.com/

Why should anyone choose to use Apptimize over a free open-source alternative?

One of the biggest reasons is probably the reason behind using any software as a service rather than doing it yourself: you can concentrate on what you do best and not spend time managing things yourself. A couple of our users switched from using open source alternatives for that reason. We've put a lot of time into our infrastructure and decoupling things so we're very resilient against outages so you don't have to. We've also put a lot of time into making our library more robust and well tested than any alternatives we've seen to date.

I'm guessing because it's better than what they could integrate and build up with an open-source alternative in a short amount of time and thus more cost effective.

I've got some karma points to burn, so here comes something.

A list of your clients would be nice to see. So that I'd know to never install any of their apps.

I hate apps that phone home. I hate apps that phone to some random analytics companies even more. I don't like companies that enable, entice and profit from this sort of behavior, a behavior that every single user would opt out from if they were asked upfront. So, unfortunately, the only thing I can wish you at this point is a speedy demise so that you'd be free to focus on projects that are less ethically questionable.

/rant

How does the anonymous gathering of statistics harm you? As an app developer, this data allows me to measurably improve my apps.

I have several free (no ads either) apps in the Android market, and even in the most trivial of apps I will include Google Analytics to receive crash reports, see what screens are most popular (so I can improve my apps where it matters most), where the app is being used (for some reason one of my apps was really popular in France, I then added a French translation and the adoption really took off), what devices are most/least popular etc. If I add a new feature, I want to know if its being used or if I wasted my time, if I need to improve the discoverability of the feature etc.

Last but not least, the usage data is really motivating to me as a developer of free apps. If I release an app, and I see it being used daily, that makes me happy and motives me to keep updating it.

As far as I know, the GA data is anonymous and only shown to the developer in aggregates.

I agree with you 100%. What it ultimately boils down to is the need for feedback.

Publishing an app is a conversation with your users instead of throwing a message in a bottle into the ocean. You need all kinds of data and metrics not only to see how well your business is doing, but also how you can help users have a better experience with your app. For better or worse, until telepathic feedback is invented, phone-home is all we have.

> As an app developer, this data allows me to measurably improve my apps.

I don't give a flying f#ck about that, sorry. This is my device and I don't like my device connecting god knows where at a whim.

As I said, if you were to tell your app users upfront that you will be tracking their usage, anonymous or not, and gave them an option to opt out, very few would not opt out. This is the exact same inherent need for privacy that is fueling current NSA debacle. It may be irrational, but it pretty damn real. People don't like to be tracked or watched.

I think that frankly you are roundly wrong. Most users just don't care that they are anonymously being tracked or monitored. Most users what good, ease to use, reliable apps that do something interesting. If it phones home to google most people just don't care. The NSA scandal was mostly met with a loud "meh?" from most people. The media is pushing it as a story but really only techies give a shit about it. I'm not saying it's right but I'm pretty sure that's the general feeling about it with the general public. Anon analytics make better apps. It's a good thing for users all the way around. Better apps == a better user experience, that doesn't happen without better data.

> Most users just don't care that they are anonymously being tracked or monitored.

Most users don't know, because you didn't tell them.

> Anon analytics make better apps.

Highly debatable.

>> Anon analytics make better apps.

> Highly debatable.

How can that be debatable? If you see 45% of crashes happen on the photo-upload screen, isn't that some kind of clue where to look? I really don't think anyone can argue that analytics are not helpful in improving app quality. Please, if anyone can even play devil's advocate on that... please do. But you better make a throwaway account 'cuz I suspect you'll be buried into oblivion and ghost-banned.

The crash report backtrace can be gathered on-demand, after asking the user's permission.

On-demand user-approved crash reporting is not usually what people mean by phone-home 'analytics'.

> But you better make a throwaway account 'cuz I suspect you'll be buried into oblivion and ghost-banned.

That's pretty rude.

You may want to consider a flip phone but even that has a GPS for 911... A/B testing is the least of your problems. If you are that concerned about privacy get off the cell network, root your phone with AOSP or your own OS and most importantly do not install any apps that you have not built. At that point consider A/B testing as it could greatly improve your user experience ;-).

Joke aside I understand your concern but this is really not the right place to start.

If he is using your app, it implies that he somewhat trusts you. I think it's Google that he doesn't trust. Using an app shouldn't require trusting a third party with personal behavior data.

You're more than likely using the Android Marketplace, with a Google account, with credit card details associated. What don't you trust?

In general, apps that "phone home" suck, but let's not lump GA or crash reporting (like BugSense, etc) in that same group.

Asking permission to record a crash is very different than silently sending user-identifiable data to Google (or elsewhere).

The data as sent to Google (or any other analytics system) contains more than enough personally identifiable meta-data as to serve as a hugely useful database for the mining information on specific individuals.

Even just an IP address and device identifier is very useful metadata for assembling sweeping user profiles.

This creates a centralized data trove of hugely valuable data (why else do you think services collect it for free?) that is useful to both commercial and government interests.

This is why personal privacy matters, and why silently 'phoning home' without first getting explicit user permission is wrong.

Seriously? How do you feel about A/B testing on websites?

He may have simply blocked all third party cookies and javascripts so it may not be a big deal to him.

Sending a request to a server is explicitly and inescapably part of requesting a web page; users explicitly grant permission for that request to be made obvious to you simply by making the request in the first place.

On the other hand, there's still an open question of what you do with that request data. Sending it to Google without permission is wrong, as it contains a slew of personally identifiable data that allows Google to build a massive profile of a specific user's behavior both across the internet, and in the real world.

I really like the flat design. Nice and subtle. Also, Optimizely reference?

Reply on news.ycombinator.com