How Twitch Learned to Make Better Predictions About Everything (2017)
hbr.org
hbr.org
Both approaches have strengths and weaknesses; the application of the appropriate tool at an appropriate time tends to have the greatest success.
Or, to put that another way: by "frequentism", the parent poster is referring to those people who believe that one thousand experiments on five people each add up to nothing, because none of them individually had enough power to draw a significant result. And by "Bayesianism", the parent poster is referring to those people who just do the five-person experiments and use whatever data they spit out, however noisy it is, because fractions of a bit of information are still more information than they had before.
Wow, everywhere I've ever worked has tried (consciously or unconsciously) and succeeded to build the exact opposite culture.
Problem needs to be a convincing and preferably very common problem of some kind. In this article it's "forecasting". And, of course, standard HR/Management practice (surveys + "impartial" statistical analysis in this case) is the way to solve the problem, but of course nobody does it right.
How to solve it ? Buy book X or, if you've got at least 100kg of money to thrown down a hole, hire consultant Y. Only 10kg of money to burn ? Visit website Z, subscribe, buy video, whatever, and get colorful pictures stating the obvious, often with audio.
For this one it's :
A = "forecasting" (really deciding future direction)
B = "forecasting tournaments"
X = https://www.amazon.com/Superforecasting-Science-Prediction-P...
Y = Philip Tetlock ( tetlock@wharton.upenn.edu )
Now, don't get me wrong. Hiring organizational consultants CAN work, of course. Getting ideas from within an organization and being frank and fair about them can bring incredible results. Having someone else come in, see the organization and tell you what's wrong can at the very least give you an idea of what's happening from other people's perspective. Maybe it can help you improve things. If you can afford it, I would advise to do it.
I'm sure this person is a capable organizational consultant, but ...
I would bail so quickly it would make my head spin if I had to work at a place where people are ever shot down for not knowing things and asking questions. The punchline is a lot of those people would tell you they're working on "the most interesting thing" and yet if you're working in a space where everyone is supposed to know all the answers... then there's nothing to learn. That doesn't sound very interesting to me.
After over 15 years and a handful of jobs, from POS enterprise software to game porting, I started to think of psychological safety as a SciFi term :)
I'll keep looking for my holy grail.
I wonder if it's something you can weed out companies for during interviews. I imagine you could iterate on some questions for the interviewer that, assuming they didn't lie, would give you a pretty good indication of this "safety" factor. Something like: "Can I go up and ask anyone a question about what they're doing?" It might rely on you speaking to real employees. I imagine if you're speaking to some very HR-y types, they might just be focused on saying "yes" to whatever question you have.
Every other question, a follow up to the substantive quotation, asked you to evaluate how sure you were in the previous answer.
So a question might be a little theory or short answer or maybe asked to write code to do a simple bloom filter. The next question asked how sure you were it had no bugs or would work if all values were 4 character strings.
Probably the second toughest interview I ever went on. But I found the idea fascinating. The idea was for the quiz to test your ability to work under pressure and be able to evaluate risks in your code.
But what was the toughest interview if that wasn't it?
The bank's questions were just brutally difficult. From low-level C++ and Java internals to algorithmic ones where you are at best hoping for a good approximation. And one very good series of questions on modeling a game where the answer involved using a stochastic matrix or monte carlo simulations.
I'm curious to know how they arrived at the 80% interval and whether that degree of certainty is optimal. One could argue that 50% would be a better choice if the goal is to improve one's ability to forecast.
I wasn't going to commit myself to a hard number if it wasn't life-or-death stuff. The company (and no company since) was worth that.
I further wondered if throwing in a point estimate as well makes sense. For things that are along the lines of "definitely X, but the sky is the limit. Well probably closer to the lower bound."
Has anyone had a similar situation? How to handle that?
I'd love to do estimates and proof them right or wrong. But bundled feature releases, marketing and season all end up in the same number of "total sales this month". So far I did not manage to make them at least track churn/retention in a detailed way.