Comment by animal_spirits | Hacker News Reader

Comment by animal_spirits | Hacker News Reader

> British soldiers in World War I were equipped with a Brodie helmet, a steel hat designed to protect its wearer from overhead blasts and shrapnel while conducting trench warfare. After its deployment, field hospitals saw an uptick in soldiers with severe head injuries.

> Because of the rise in injuries, British command considered going back to the drawing board with the helmet’s design. Fortunately, a statistician pointed out that the dramatic rise in hospital cases was because people were surviving injuries that previously would have killed them—before the introduction of steel the British Army used felt or leather as headwear material.

I've seen this same story except with warplanes during the War. Story goes that an allied air force tried to improve the percentage of planes that would return from a bombing raid, so they inspected returning planes, found the places where they had holes in them and added extra layers of steel to those areas for the next bomb run; but this had no improvement on the percentage of planes that returned.

That is until a "statistician" realized that all of the places on the plane that they found holes in were actually parts of the plane that could get hit and survive and return. They then started adding extra steel to the parts of the planes they found no holes or damage assuming that if those parts were hit the plane would get shot down and not return. After this change they started seeing a dramatic increase in the amount of planes returning from a raid.

Does anyone know the true source of these stories?

Regardless, it really opened my eyes to how changing your perspective on the cause of the problem can help find the best solution, and to this day I still think of this story when solving a problem.

We apply this kind of thinking at work: early on we focused all our energy fixing user problems or adding features that were requested. We realised it is actually a small number of users who volunteer to report to us what's wrong or missing (outside of regular crash/feature tracking) - they are the ones that care enough to make an effort, and it's a small percentage. So we started campaigning to engage users we never heard from, to understand what problems they had that we weren't solving, to get them energised enough to report to us. It's been very successful, engagement has gone up significantly.

Did those users actually report different issues altogether from the original ones, or did they just report more of the same kinds of issues?

Yes there is a large set of venn diagrams we weren't connecting that it gives us pretty easy access to greater coverage. The harder part becomes messaging and marketing as we target more use cases.

What mechanisms did you use to get users providing feedback? Popups asking or something?

Good old email segmenting. It's become a real powerhouse for us. Figure out what groups of people you have, talk with a few to figure out what they want, build it and target those who match. We're not a startup however so this is more of a scale up strategy or if you already have access to a large number of users who trust you and want just send your emails to spam.

This makes sense. The sort of users that leave any kind of feedback or even hang around long enough for you to gather statistics are those who think the product is pretty good, but there are just these small issues that they hope will get fixed. The users who think your product is complete crap try it, stop using it very early and never leave feedback or statistics for you to analyze.

Way back in the day I was a lone IT guy in a distributor/wholesaler, and part of that was creating line of business apps to do some real heavy lifting, extending existing ones, and creating utilities to tie other apps together. In a tiny company of maybe 40 people, where even an underpaid IT guy was still an extravagance (but was adding huge value).

Periodically I'd ask to shadow someone for a day or two. At some points, even asking if I could do their task and have them watch me to confirm I was doing it correctly.

What you would learn from this was amazing. All sorts of inefficiencies that people would quietly accept, because it was still way better than the previous state.

Then a week or two later I'd roll out the update, and get huge thank yous from accounting, or sales, or the warehouse because I completely trivialized some previous common task. What used to take 45 seconds now takes 2 because all the work is done for them 99.9% of the time and they just need to confirm it's correct (or correct the odd edge case). The computer became this increasingly magical tool they loved more and more.

The number of hacks and workarounds that users will come up with without complaining to the dev team is always astounding. One time I shadowed someone who was using one of our internal tools that spit out pdfs, and at some point along the line he realized he wanted something extra included on them but didn't think to contact the dev team. As part of his daily work flow, he was taking data from one of the program's temp files and editing the pdf to add it. He had apparently been doing this for about a year. When I saw this, I spent 10 minutes updating the report template to add it and ended up receiving thank you emails from people I'd never talked to.

So yes, find a user and shadow them.

What are some tactics you use to get these customers amped up?

Taking the train over the Rockies they tell a story about Westinghouse[1] inverting the logic on the pneumatic brakes. When first introduced they failed open, meaning the train could move. This was convenient on the flats, but if the train was stopped on a grade and the pressure failed... down the hill you go. So Westinghouse made them fail closed: no pressure, no rolling down the hill. IIRC it saved lives.

[1] https://en.wikipedia.org/wiki/George_Westinghouse

You are thinking of Abraham Wald: https://en.wikipedia.org/wiki/Abraham_Wald

Note that the Wald story may not have actually happened: http://www.ams.org/publicoutreach/feature-column/fc-2016-06

(But yes, Abraham Wald is the statistician that the story is about.)

yes and here's some excellent metaresearch on the "measles plane" https://counting.substack.com/p/its-that-ds-meme-plane-with-...

Aaaaaactually... in the postscript he sort of walks back everything. But since the article has mathematical and educational merit on its own, I guess he maintained it.

"My indignation at how the internet dealt with Wald's work was overblown. Stephen Stigler (son of George, and a statistician at the University of Chicago) called my attention to a note by W. Allen Wallis himself in which he mentions Wald's work explicitly in connection with survivorship bias. Wallis' original article in the Journal of the American Statistical Association was followed by two very brief comments and then by a further 'rejoinder' of a bit more than one page. Towards the end of it he says, "The military was inclined to provide protection for those parts that on returning planes showed the most hits. Wald assumed, on good evidence, that hits in combat were uniformly distributed over the planes. It follows that hits on the more vulnerable parts were less likely to be found on returning planes than hits on the less vulnerable parts, since planes receiving hits on the more vulnerable parts were less likely to return to provide data. From these premises, he devised methods for estimating vulnerability of various parts."

Amazing article btw. Thanks. This is what makes HN so unique.

Yes, thank you. Now I know where this came from!

https://en.wikipedia.org/wiki/Survivorship_bias#In_the_milit...

That plane with the red Measels spots has become a joke meme on twitter. Here's the Wikipedia page on it to start you off on your deep dive:

https://en.wikipedia.org/wiki/Survivorship_bias

old tech blog about improving youtube page weight -- it paradoxically worsened their long tail stats

once they sliced it geographically, they found that the increase was all in places like siberia where previously the site was unusable

https://blog.chriszacharias.com/page-weight-matters

I always feel these stories are nonsense, they no one can be that stupid. Then I think about the people I’ve dealt with over past 20 years. And I fully believe these scenarios.

> That is until a "statistician" realized that all of the places on the plane that they found holes in were actually parts of the plane that could get hit and survive and return. They then started adding extra steel to the parts of the planes they found no holes or damage assuming that if those parts were hit the plane would get shot down and not return. After this change they started seeing a dramatic increase in the amount of planes returning from a raid.

This is highly likely to be grade A BS because it would be quite obvious to literally everyone flying an aircraft. It only make sense because the reader are not given a full minute to think about this.

Reply on news.ycombinator.com