Plugtest
en.wikipedia.org
en.wikipedia.org
I tried to find a source for the story but I couldn't find one. I think I read it in The Innovators by Walter Isaacson, but I can't remember it exactly - maybe I'm misremembering it.
The issue I've always found with testing on human users is their willingness to ignore deficiencies.
There is a strong benefit for people "just getting on with it" and "working around" a problem; but it seems like many people are not very good at identifying when a problem is worth solving.
An analogous anecdote I'm aware of was a small in-house call-centre team being moved to digital playbooks (vs those A5 ring bound / tabbed sets). While time-to-resolve each issue got lower and time-to-adapt to new workflows a lot faster, many tracked metrics got worse. Suddenly they were picking up fewer calls, time to answer was increasing, job-satisfaction decreased.
It turns out the new system was forcing the team to be sitting at their desk. In a small and busy multi-team office these people were usually doing all kinds of other small things - be it making coffee, liaising with other teams, etc. - whatever it was they weren't always at their desk. They did have wireless handsets however, so they used to answer the phone, start the call from memory, and move to use the nearest playbook.
The big issue with the digital playbooks was needing to "follow along", you can't reasonably start it away from your desk.
The team knew this was an issue very quickly, it was quite fixable, but they sucked it up for months.
Another story is how the first product made by Jobs and Wozniak at Apple was a hacked phone that could be used to make international calls for free. They called the Vatican and pretended to be Henry Kissinger wanting to speak to the pope - they noticed and promptly hung up before they reached him
> Telephone company executives wondered whether the standard cord, then about three feet long, might be shortened. Mr. Karlin’s staff stole into colleagues’ offices every three days and covertly shortened their phone cords, an inch at time. No one noticed, they found, until the cords had lost an entire foot.
https://www.nytimes.com/2013/02/09/business/john-e-karlin-wh...
We had a hierarchical storage management driver, and getting files released to remote storage and keeping them released until the users actually wanted them back was helped greatly by actually being able to talk to the AV vendors and understand their criteria for when to scan.
At the time, a lot of AV filter drivers were legacy model drivers, which led to some serious shenanigans getting a second minifilter installed above the AV layer in the stack to distinguish between user and AV generated reads.
https://learn.microsoft.com/en-us/windows-hardware/drivers/i...
I've always wanted an excuse to write a minifilter or similar and solve a real problem and totally not at all hose my system's performance. Windows kernel development calls to me for no good reason.
https://operations.nfl.com/gameday/behind-the-scenes/nfl-eve...
https://www.mixonline.com/live-sound/tackling-rf-for-the-sup...