Hacker News Reader

Comment by jordansmithnz | Hacker News Reader

Solo dev for an app with hundreds of thousands of users. These are the things I do that ensure smooth releases:

• run a beta test group. These users consist mostly of users that have emailed in with feature suggestions or bugs. I ask them if they’d like to join the beta (e.g. they can check their bug is fixed). I frequently get great feedback from a few of these users.

• fiverr.com. Shell out $100 spread across a range of testers, and give them all slightly different briefings. Some feedback is terrible but there’s some gold there that’s really worth far more than $100.

• learn how to test your own product. It’s hard to do well, but you can learn. It’s a valuable skill that can really set you apart in a regular engineering job too.

• used phased releases. Pause the release at 1%, give it a few days for any minor issues to roll in and fix these before releasing to 100%.

View on HN

Also a "Solo dev for an app with hundreds of thousands of users"

I just test everything myself. On one OS (Mac). The worst part is having to run VMWare with older OS versions sometimes, but not usually.

I find that bugs are almost never OS specific, and are rather just logic errors.

I find that having other people test my code adds little value, as I can literally just test it myself in the OS that I write it in most the time.

I know that this goes against all advice that you read, and all the "common knowledge" that is out there... but I've been doing this for 15 years and have got over 1 million users in total probably, and I almost never get support emails regarding bug reports. So, in my experience, I really find that everything I have heard is wrong. I realize that other people may have other experiences but found this point of reference interesting.

Experience building software can lower the testing requirements.

I find the degree of testing needed can be relative to the difficulty of the problem, the unnecessary complexity of the solution, and the average skill of the developers working on all 3.

Clever (simple) architecture will often beat clever programming in the long run.

> Experience building software can lower the testing requirements.

This is a good point. The longer I've been at this, the less often, "this is likely to work" is a thought that I let go unchallenged.

"This is totally going to work" is only said when I am completely sure that it is going to fail catastrophically. If anything works I feel mild distrust with my code from that point forward.

Knowing the difference between thinking less about yourself vs less of yourself is important when taking your work seriously but not taking yourself too seriously.

I have a healthy doubt of my own work that seems to end up making it work well first by keeping it as simple as possible.

Keeping the mindset of an innocent beginner is critical as a developer with experience. Everything is easy in the beginning, not the long term - I like to see what code bases people have worked on for many years instead of being able to neatly start from scratch.

Yep. I try not to get caught up in worrying by adding lots of tests and making my code easy to change. When I find an old crusty piece of code I actually have the most fun because it takes me completely immersing myself in it to find all the crap floating around.

I think it's only fair, as a developer, to live in a state of constant fear and paranoia.

(joking)

Absolutely. It doesn't mean testing isn't required - it's absolutely the case when something is business critical, or average / inexperienced devs are painting unicorns and rainbows everywhere.

I have a healthy skepticism of my own work, it keeps hubris in check. This combined with a preference to keep things simple (to come back to later and to let others join easily), I find encourages everyone to build a code base they want to work on long term.

you don't mention or allude to automated testing. You don't have any? (i.e. 100% test coverage except flip the 1?)

I think you misread the question - the question is about "user testing" not "QA Testing".

> used phased releases. Pause the release at 1%

Could you expound on this? How do you do that? (Related: How are you distributing? Through an app store?)

There's an option to do this on both Google Play and App Store. It's called staged rollouts[1] in Google Play and phased release[2] for App Store.

[1]: https://support.google.com/googleplay/android-developer/answ...

[2]: https://stackoverflow.com/questions/45073264/what-happens-wh...

Solo dev and 100ks of users? Wow.

What is your tech stack? What is your code-deploy process?

Probably a mobile app, and probably deployed via the platform tools. They have options for beta releases which users can opt-in themselves, and also phased releases are supported out of the box. Just my guess.

It's probably his mobile app called Class Timetable

https://news.ycombinator.com/item?id=14542045

> fiverr.com. Shell out $100 spread across a range of testers, and give them all slightly different briefings. Some feedback is terrible but there’s some gold there that’s really worth far more than $100.

Also check out Cent (https://beta.cent.co) - $20 in Ether can get you a pretty good set of responses. I am a cofounder - feel free to ama here or at cameron@cent.co

So you want most of the work to happen for free? I really hope your product fails. This is even worse than mechanical turk.

The way Cent works is instead of having 100 people earn 20¢ (for a $20 bounty), $20 gets distributed to the top 10 responses. Respondents are encouraged to be thoughtful because the community votes on which 10 responses get the bounty.

Many responses don't get paid as a result but making it competitive adds more value for the bounty-asker and makes the whole thing more fun in general.

What about this do you find so terrible?

Great gain for the bounty-asker, great loss for everyone > 10. Big win-lose gap. Just plain exploitation of the workforce. I don't see the fun in that.

This sounds like very actionable advice. Thanks for sharing the info, it's very helpful.

Reply on news.ycombinator.com