Ask HN: Solo app devs, how do you do user testing?
Enlisting the help of friends and family can only get you so far, so how do you gather quality feedback pre/post-launch, without shelling out for a professional agency to do the testing?
• run a beta test group. These users consist mostly of users that have emailed in with feature suggestions or bugs. I ask them if they’d like to join the beta (e.g. they can check their bug is fixed). I frequently get great feedback from a few of these users.
• fiverr.com. Shell out $100 spread across a range of testers, and give them all slightly different briefings. Some feedback is terrible but there’s some gold there that’s really worth far more than $100.
• learn how to test your own product. It’s hard to do well, but you can learn. It’s a valuable skill that can really set you apart in a regular engineering job too.
• used phased releases. Pause the release at 1%, give it a few days for any minor issues to roll in and fix these before releasing to 100%.
Could you expound on this? How do you do that? (Related: How are you distributing? Through an app store?)
[1]: https://support.google.com/googleplay/android-developer/answ...
[2]: https://stackoverflow.com/questions/45073264/what-happens-wh...
What is your tech stack? What is your code-deploy process?
I just test everything myself. On one OS (Mac). The worst part is having to run VMWare with older OS versions sometimes, but not usually.
I find that bugs are almost never OS specific, and are rather just logic errors.
I find that having other people test my code adds little value, as I can literally just test it myself in the OS that I write it in most the time.
I know that this goes against all advice that you read, and all the "common knowledge" that is out there... but I've been doing this for 15 years and have got over 1 million users in total probably, and I almost never get support emails regarding bug reports. So, in my experience, I really find that everything I have heard is wrong. I realize that other people may have other experiences but found this point of reference interesting.
I find the degree of testing needed can be relative to the difficulty of the problem, the unnecessary complexity of the solution, and the average skill of the developers working on all 3.
Clever (simple) architecture will often beat clever programming in the long run.
This is a good point. The longer I've been at this, the less often, "this is likely to work" is a thought that I let go unchallenged.
I have a healthy doubt of my own work that seems to end up making it work well first by keeping it as simple as possible.
Keeping the mindset of an innocent beginner is critical as a developer with experience. Everything is easy in the beginning, not the long term - I like to see what code bases people have worked on for many years instead of being able to neatly start from scratch.
(joking)
I have a healthy skepticism of my own work, it keeps hubris in check. This combined with a preference to keep things simple (to come back to later and to let others join easily), I find encourages everyone to build a code base they want to work on long term.
Also check out Cent (https://beta.cent.co) - $20 in Ether can get you a pretty good set of responses. I am a cofounder - feel free to ama here or at cameron@cent.co
Many responses don't get paid as a result but making it competitive adds more value for the bounty-asker and makes the whole thing more fun in general.
What about this do you find so terrible?
Beer.
You think I'm kidding but if you go to a bar, there are lots of people who will test whatever you put in front of them for a free drink. If you want to keep your costs down even further then go during happy hour.
Is that "user testing" no... not in the sense we think of it today. Is it testing of the usability of your product: yes it is. Im guessing that most apps issues that are more likely the latter and not the former. Basic things that you think are clear that really aren't (your too close to the problem and the solution). If you explain, point and guide through the same aspect of your feature 10 times in 10 tests people might not get it! These are the problems you might want to solve, dumb it down, change it up, make it better.
Does this work? Yes, as long as you can moderate. If you can you be neutral and friendly and avoid "traps" that will alter your results. Saying things like "MY app" or reacting personally will alter your results, and learning when to "guide them" through a frustrating point and when to let them puzzle through it is a bit of an art (this too offers clues if your guiding people at the same place). For me personally a drink or two helps ME in the process...
I have also discovered that this process works very well with pairs of people. Two friends already have a rapport and will happily talk to each other to try to get through something problematic. One can encourage or discourage the other and their dialog can be enlightening. Letting them play together with your app and drink free drinks can give you a LOT more insight than the next three individuals testing. Anything more than two people becomes problematic so look to avoid large groups.
So probably how they will be when actually using your app? I do agree though in a reproduce-ability/data-integrity standpoint.
If I had to pick one thing to measure, I'd be more inclined to measure positive versus negative fucks (I'm not joking), but honestly, it's more about the experience and the totally raw, sometimes terrifyingly honest feedback I hear. Numbers aren't useful because the most useful sessions go way off the rails.
I am OP, and it depends.
Im old enough that I have done this with paper prototypes, and it is shockingly effective.
Now most of the code I write is compartmentalized enough that with a bit of effort I can come up with a one off digital artifact. Just rip out the chunk of your app that you want to test and present ONLY that. Spend a day building ugly shims early enough in the process and you might make some dramatic changes!
> Be aware that you are not controlling the environment and the users may be distracted by outside factors. Which may affect some of your metrics (time to perform tasks, success rate...)
Even with those things going on your going to be shocked at not only what issues come up but how frequently. That thing you did that you thought was simple and sane, no one gets it. Do more than half of your test subjects comment on the same thing (good, bad, both) then you have a problem.
The key isn't controlling the key is listening...
I also want to state that a slighty drunk, distracted bar patron, getting through your process without quitting, telling you your product is shit, or giving up is a positive sign that in the real world people who are more motivated than "beer" are going to have a fairly positive and expedient experience.
Doing this step first is a good way to avoid some of the worst case issues.
2. Release the beta, mark it as such and ask people for feedback after they used the product for a bit. Listen to the feedback, act on what's relevant, keep improving the beta. Give kudos where they are due.
Have public forum, actively participate in all threads. Many people look at how active the support forum is before trying the goods or leaving any feedback. So the more active it is, the better. Also, moderate it aggressively to trim junk and trivialities.
3. Once out of the beta, don't forget to genereously reward everyone who chipped in. Ideally with a special gift, like a specially tagged lifetime premium account, a handful of most expensive licenses, etc. The worst thing to do is to offer a discount on a production version - this comes across as a cheap and greedy move.
There are no better user testers than your users, and the risk is only moderate. I could always pull the server down and restore the previous version while trying to fix whatever I broke in the deployment.
But if you aren't doing SaaS, user testing as a solo developer is not really feasible. The best user testing I can think of that doesn't "cost" is a Beta/RC Program for your users to volunteer to get potentially broke software for a discount or just early feature releases. For this to work though, you have to have enough users.
That's why I have always used a 3rd party system to collect actual usage analytics combined with server logs.
When I have a decent, bug free mvp of the next version I'm going to fork out for usertesting.com
I also use ACRA and Tracepot to capture crashes and anomalous events. Users can also email me a debug log along with their current app settings. This has evolved over a 5 or 6 year period.
What's your goal? Do you want to test whether your update works correctly on many devices? Do you want to get ideas for new features? How many users do you have?
I have about 10,000 active users. I don't do much testing. I test myself on an Android device and an iPhone. Since my app is built with Ionic and based on Cordova, it works anyways and real crashes are not to expected.
Feedback is easy: I just wait until people write me an email. It's funny: Sometimes, you don't hear about a feature request or some issue for months. Then 2-3 people write you within a day. Then nothing again.
Back when I started, I implemented a lot of features. Now I basically reject almost every feature request, unless it comes up extremely often in emails. I sometimes take notes, but most of the time, I type a quick reply, thanking for the suggestion, archive the email and only try to get a feeling for what people want.
Obviously, this has its limits, but since I know my core users and what they want, it works for me.
If there's a serious bug, people will let you know soon enough.
I was thinking about just opening up on the app store as well, and just sit back and wait for reviews or emails, but that seems risky, since potentially bad reviews are forever. For a paid app at least, that can have a pretty serious impact on possible uptake. Having at least some level on confidence, that the app is usable and stable before general availability, is the safe play I think. To that affect, closed testing seems the only viable option.
Your ideal tester is someone who cares enough about your product to use it.
I also wouldn't even start developing the product unless I had a decent sized list of people who really want it.
In which case, testing it is easy. Just ask those people to test drive your app during development. If your app solves a real problem they will happily do it for free since it solves their problem, but you could also give them perks like giving them 2 months of your service for free when it launches.
Your users will naturally give you feedback if things are broken, so that handles post-launch real world testing. Of course you'll want to supplement that with heaps of automated tests too.
If you use heatmaps and other analytical tracking tools you can see how users interact with your app.
That's all you really need to see why a user isn't using a feature, struggling with the UI or happen to not see something.
An agency would just have X number of paid employees systematically going through your UI and giving you their opinions on what they think is best, but that might not align with what your real users are doing.
You need to be make it as easy as possible to run your user test, because a tester will give up on roadblocks that take more than 60 seconds to solve.
I was developing a language learning product, and so needed testers who were interested in language X. In the end, the people I found using the filter [Interested in X language, native English] weren't actually interested in language X. Rather, they came across as people who had accepted the filter to make an extra buck.
However, it was good enough for a random internet trial. If I wanted better results I had to do it in person (which I also recommend).
- don't "burn" beta testers. show a prototype to one or two people, and don't show it to more people before you fixed the issues that have come up. otherwise you just get duplicate feedback
- ask people personally. Emails to a group of people are ignored by 99% of recipients
- feel free to ignore advise from testers. everyone always wants to give advise, but that is irrelevant. the important part is: did the tester manage to complete the task?
- do not ignore people struggling with your app. the natural reaction to people failing to use your app is to think they are stupid. they aren't. most of the time your ui is stupid and you need to fix it. it doesn't matter how brilliant you think the ui is, if testers can't use it, change it.
- sometimes the answer is that what you built doesn't work and you just wasted a lot of time and you need to throw it away and start over. do not ignore this possibility
The above is what you get from usertesting.com and other sites but pay up the nose($50 a test) vs. $10 or less dollars per test.
If it’s a downloadable app Just spin up a web UI/UX and have the turkers follow same steps but have them change the browser to say iPhone 6/7/8 view.
This doesn't mean logging in, trying all the features and playing with it for an hour - you have to really use it with your data on a daily basis over a long period of time. You'll come across a lot of bugs and start logging feature requests yourself.
Once you can use it without issues [and you'll need to be careful to split the wishlist features you come up with, against the actual bugs which need to be fixed], then you can get others to try it.
Best way to get user testing from others?
- email your users and ask them if they have had any issues, or want any new features.
- ask people to give your product a quick test - for example, https://www.lifepim.com is my new webapp to easily manage your personal information [shamless plug/warning - notes and tasks work well, contacts needs work, and calendar is not pretty]
- pay particular attention to requests from users who pay for your product. Everyone wants software to work in a certain way, but when someone pays, it can be a good indicator to listen to what they want. If they say something isn't working, make it a priority to investigate and address this.
* http://www.trymyui.com/
* https://www.usertesting.com/
* https://Whatusersdo.com/
* https://Userlytics.com/
In the next couple of weeks I'm due to release a boxing game (shameless plug: leatherthegame.com), and other than myself my only tester has been my cousin.
Having worked on the project for over 3 years I'm aware that there are areas where I "can't see the wood for the trees", but I am confident in my own knowledge, and only wanted an external viewpoint to see how a noob would see things from afresh (both in terms of not having used the app before and also him being only a lite-casual boxing fan, so he wasn't up-to-speed with some terminology).
I know full well that once its released the proper boxing fans will get in touch, and I'll change things post-release from that feedback, so my strategy is very much a post-launch end user feedback one. In an ideal world I would have had more friends who could help me test it, but there's only so many opinions I can factor in pre-launch, and managing their experiences and feedback would have been a mini-project to manage purely by itself. When already time-constrained with launch schedules, app store listings and last minute bug hunts and device optimisation I couldn't spare the time to manage a phalanx of testers.
This has been a passion project/scratch-my-own-itch project, which is why I feel more secure not having many outside opinions. If you're building something more strategic, to fill a target niche in a specific market say, then my approach will DEFINITELY NOT WORK. In this case you'd be best off firing out a message on LinkedIn or something like that, asking for would-be best testers.
The form served 2 purposes:
1) It gave me the ability to balance different types of users. Hobbyists vs. professionals. People in this specialty vs. that specialty. etc.
2) It weeded out people who just wanted something free and weren't interested in giving feedback. Seriously, a simple form requiring a name, an email, and the answer to 3-5 questions helped get rid of the freeloaders. But don't make it too onerous or you won't end up with any testers.
I'd then usually give a free license for the released product to anyone who provided useful feedback.
In fact this worked so well that I am currently building a new project around this. A site that lets startups gather feedback from people who are interested in trying out new sites and giving feedback.
It is currently in private beta with about 20 startups. If you want an invite, feel free to send me an email.
I know this isn't the answer most people want to hear, but no one tells the truth like live users.
Eventually, when it's easier to pay, some users will start leaving money instead if feedback. At some point I'll raise prices a bit and stop the feedback for service system.
I've done this for:
https://projectpiglet.com/ and https://easy-a.net/
And a few others, it's always fairly successful - assuming you have a decent landing page and you can advertise the site in your target market for cheap / free.
A few times now, I have successfully got my first few hundred users by offering a lifetime discount and positioning it as a limited access beta testing program.
Got lots of feedback, and most users kept their accounts for a very long time, even if they weren't using the software, just do they wouldn't lose their discount.
I start out with manual tests.
If I have a bug at any part of the code I add a unit test.
When it gets too big and starts taking too long then I start writing e2e tests.
Its true that unit-tests ensure quality software. But this quality is a waste if real user's can't figure it out.
good price, cheap compared to a dev’s salary, or compared to my time, paying less than $500 per month.
do 5-second tests, usability, card sorting, path testing, all ux and ui testing for greater chance of success