Hacker News Reader

It might even be worse still since this model (seems I may be wrong) to assume that the probability of a usability bug is constant, it might be that the share of bugs discovered by users is skewed towards the first few such that the first user finds more than the formula would predict.

It's certainly been my observation that cynical developers who test things as they go by deliberately putting in silly things into stuff they just wrote seem to get hung up less in testing.

I mean the system I inherited at work the first thing I did when I got an instance spun up was put in a negative value in the quote line quantity (which immediately broke..well almost everything) then decimal values in quantity fields where only integers made sense, then text in number fields and so on each time breaking something in a new and interesting way.

Sometimes I think it's hard not to be cynical about enterprise systems.

My old lecturer (somewhat pithily) "Almost all the testing in the world means nothing compared to 15 minutes in the hands of the 17 year old office junior"

One of my friends was literally hired as an intern to try to break software last year. He loved it, and found a ton of bugs, which was really helpful to the company- they eventually paid him a $1,000 bonus for his help over the summer.

It sounds like this guys employer has taken first step to inventing QA.

There's whole classes of highly paid engineers whose job is to do this. But they work for old fashioned, boring, companies.

The problem is QA tends to get to scripted. You need to exercise each corner of the product, thus you test A, B, C in that order - so you never find cases where testing C, A, B breaks, or any other permutation. (to be fair with any complexity it is impossible to test all permutations)

That's one part of it.

Another part is hiring a breed of test engineers who like breaking stuff and have a knack for it.

We call it exploratory testing.

I can believe it, the best person to test software for low hanging fruit of the "What happens if I do this thing that no sane person who knows anything about what the software is supposed to do would do?".

It's one of the reasons why I don't trust myself to test things fully, we write the software with all sorts of assumptions in our heads and subconsciously steer away from doing silly things - in that context it's really difficult to aim at a point a zero-knowledge user would hit.

I'm been building an API Security Scanner, and one of the things it does is just fuzz every endpoint with garbage in each parameter to look for stacktraces, errors, etc.

Moreso than any of the security tests I've written, that fuzzing has broken every enterprise API our customers have thrown at it.

Even a sane person can find a lot if they never touched the implementation of a specific feature.

On one of my teams, we had a hard rule where all features must be tested by 2 other team members (and if multiple people worked on a feature, none of them can be the testers). Something like every other case found something questionable, if not an outright broken edge case, that the developer(s) completely missed.

I remember I implemented 2FA for a previous company, and our QA person managed to lock himself out of the account while enabling it. I asked him how that's possible, and he said "well I went to enable 2FA, it gave me the recovery codes, then it asked for a 2FA code so I entered one of the recovery ones, but now I ran out".

I thought this was so dumb it was brilliant QA.

I have a friend who does this for free. Every single thing I've ever shown him (both my software and that of others) breaks as soon as he gets his hands on it.

He's the sort to dig as far into the internals of things as he can and then start messing with it (he implemented partial function application for C, for example [1]).

My poor prototypes never stood a chance.

[1] https://github.com/zwimer/C-bind

That's so enjoyable to do. :D

I bombed a job interview by writing a piece of demo software that failed all of your "throw it garbage" tests. I couldn't have hired a professor to teach me a more useful lesson!

Mine was brilliant, not a famous academic or anything and it was a college not a university but he'd written production systems in the 80's for telecomms and massive supermarkets so as an instructor for the real world really hard to beat - he taught me all sorts of things that stuck some of them I didn't understand at the time but 20 years later they make much more sense :).

That seems like a brutally unfair interview practice unless you were told in advance they’d be doing that.

They gave me a small project to do, and told me to do it as though I were building it for a customer. I think that was warning enough that it ought to gracefully handle bad input — any real-world program needs to do the same.

To be honest, unless they explicitly discussed this with you before, or went through it with you afterwards, I'm with empath75 on this one. We used to do something like that at $former_workplace and every once in a while, a candidate would come up with a program that didn't validate (most of) the input or failed in similar trivial ways.

It turned out that some of them, indeed, simply didn't care -- and didn't know, either. We'd explain what the problem was and they'd shrug or say they'd seen <big name software> breaking like that, too, you fix it if it turns out someone actually breaks it.

Others, however, would skip it so that they could focus on stuff that was more complicated or more relevant. They'd validate one set of inputs, just to show that they know it needs to be done and can do it, but not everything. Or they'd throw in a comment like //TODO: Validate this by <insert validation methods here>. Most of the time we'd just ask them to talk us through some of the validations -- and most of the time they could write them on the spot.

You could argue that this is very relevant in real life, and that even if it weren't, what's relevant is the interviewer's choice, not the candidate's (although tbh the latter is one of the reasons why tech interviews suck so much).

But at the end of the day it is an interview setting, not a real-life setting, no matter how much you try to make it seem otherwise. At the end of the day, the people doing it are still young candidates trying to impress their interviewers, not company employees working on a company project under the supervision of tech leads. You don't end up with much useful data unless you allow for some flexibility in this sort of experiment.

+1 but several of your points are incorrect or need caveats.

Your 1st point isn't correct, in that you will learn interesting things from 1m users that you won't from 15. The thing is that the 15 will tell you why, whereas the 1m won't unless you ask precisely the right question (which is a problem in itself). It basically takes experience here... (And this is something an AI may eventually become very good at.)

Your 3rd and 4th point aren't correct, in that given some sampling error you may very well find that what 15 people don't like 1m will.

I'm in full agreement with you in principle, in the sense that I firmly hold that far too many start-ups do quantitative without qualitative and end up with the wrong conclusions; but you can't just wave away quantitative data like that.

Re 1st point: OP was referring specifically to usability learnings

I'd argue that usability also must include accessibility and I highly doubt that in your 15 user sample set you will cover the deaf, the (color) blind, the dyslexic, the paralyzed, the epileptic and various other groups of people with special accessibility requirements.

Or, for an American developer, what are the odds that your 15-user sample will find usability problems having to do with your icons only making sense to someone in the US? There's more to localization than just translation. Or if you have a product that does voice recognition, what range of accents are you covering with those 15 users?

That's OK, we nowadays use icons that don't make sense to anyone at all. I mean how would you know that three horizontal lines was a "Hamburger menu", and if you do know that, why would you want a Hamburger?

Actually the icon looks like three menu items. It is a pretty good icon for signaling that it will open a menu. The hamburger nomenclature seems to be an ex pos facto name for something that rightfully should be called the menu icon.

> the icon looks like three menu items

It's a nondescript icon made of three horizontal bars, it looks like literally anything that comes in threes. My mom calls it the pancake button and my fourteen year-old nephew used to call it the button with something that looks like a fork but without a handle until he switched to the "meh" button once he became a nihilist (teenagers do that stuff sometimes). A menu with three items is very likely to be among the last things that crosses a non-techie's mind.

At this point it's been used enough that anyone with enough exposure to electronics knows what it does, but it's hardly a better choice than the "File" menu. The point of making something intuitive, as opposed to explicit (i.e. by using a symbol as opposed to spelling out) is kindda missed if you need to "well actually" it and explain why it means whatever it means.

I disagree. It took around two years of seeing hamburger menus before it clicked that it was a common symbol for a menu. Its getting worse and seems to be getting replaced by 3 vertical dots now.

I just think of the 3 vertical dots as vertical ellipsis. I often use vertical ellipsis when making a list that continues. e.g.

— History of Empire and Colonialism

— History of Religion and Society

— History of Race, Gender, and Power

⋮

— History of War and Society

So it naturally means "there is more stuff here" to me. YMMV

In your context it does make sense. As a menu button it doesn't (to me at least).

I can only speak for myself, but for me who grew up with C64, Windows 3.1, 95, 98, Linux, Windows XP more Linux etc that icon still didn't make sense until I got it explained.

Then again, the magnifykng glass icon for search didn't make sense either until I read the docs but at least back then people made docs and kept the UI stable enough so that it made sense to learn it.

I still remember fondly being good with OLE (Object Linking and Embedding) in a Microsoft Works.

The old days seemed so much more intuitive when there was a crappy low res icon with text underneath.

Not sure if you are sarcastic and in what way, but if I understand you correctly you think we are better off now.

I think we would be better off if we kept the best from both: the predictability and discoverability of the past with the niceness of today.

Why should we have to choose between nice and usable?

No I think that having a bit of descriptive text below an icon was far more useful than far prettier world of nicely rounded corners on hamburger menus that we have today.

Usability should include accessibility when possible, but there is no must here. There are lots of tools that need usability testing that have a sufficiently small team and user-base to make accessibility a waste of time and resources.

Indeed, on teams and products of any size there is always a decision to be made about which accessibility requirements will be met and to what degree.

I would make the separate argument that what you need with accessibility requirements is separate testing expirements for each one. You don't need 100 testers to to make sure you have some special accessibility requirements, e.g. you need 15 normal vision testers and 15 impared vision testers.

> but you can't just wave away quantitative data like that

quantity has a quality of its own

This may be true for software used when working alone, but for software used by groups, you are not going to figure out group effects without testing with different group sizes. Consider the differences testing a game with 1, 2, 5, 100, or 1000 users.

There's usability testing and there's scale testing.

Basic UI/UX will be significantly revealed be small-group testing.

Don't let the perfect be the enemy of the good.

So fifteen groups, then?

Agree, as the massive datasets are telling you more about what people want instead how the product works.

That’s a big distinction, as when you look at a million people, 80% are are either the same, or don’t really understand what they are doing at all.

Modern designers usually assume that people are dolts and need simplicity. Optimal in my opinion is providing quality on-boarding and understanding behavior of someone who knows what they are doing as well.

I definitely think there is a learnable aspect to design. Throwing data at the problem is expensive and does not create many deep insights. But focusing on communication and how the product communicates a coherent philosophy through features and UX yields a more systematic, standalone framework to test hypotheses against, something which data can further shape. It's more important to find "must" and "cannot" boundary conditions than preferences, and it's also easier to see if you are satisfying those conditions.

My formula is therefore: Pick a few principles to apply to the solution you're making, make sure as best you can that they agree with each other. Test to see if potential users agree with those principles. Select features based on the resulting design framework. Then develop the product as a way of discovering more about this framework.

Lots of companies are doing this implicitly through their founding teams and hiring practices: they just end up with culture that values certain principles, so they get upheld at every meeting and the results end up in the product. But it can also be communicated and imposed more explicitly, and I think that's where design becomes more visible in the process.

It’s not exclusive, usability is a duality. At the micro and macro levels. At the micro level you want to watch a couple of individual users and watch whether every little thing added to the UI makes sense to them. Whether the product solves their problem.

If micro fails, macro will most likely fail.

At the macro level you’re tracking higher level metrics like how many users engage with the different parts of the UI, where the fallout is? What is the retention of users, what do power users do differently to new users e.t.c

This kind of wholistic view about the product at micro and macro levels gives you a much better understanding about the bottle necks in the product.

If you have 1,000,000 users you need to ask very different questions, so you better learn different things.

At 15 users your training budget is to teach everybody how to use your mess of a UI is cheaper than the cost of fixing the mess of a UI. At 1,000,000 users your cost of training is higher, but you also have more users to spread the cost of preparing the training among.

My impression is that most people instinctively assume that to get accurate results from sampling, you need a sample size proportional to the population, and that isn't generally the case.

Maybe for many purposes, 5 won't cut it and you need 100 or 1,000, but you don't need 1%.