Open-sourcing Riskquant, a library for quantifying risk
netflixtechblog.com
netflixtechblog.com
Both books are worth reading; Hubbard's influence on FAIR is noticeable and positive. FAIR has the advantage that it comes with a fairly built-out ontology for assembling data or estimates. The OP touches on the top level (Loss Event Magnitude and Loss Event Frequency), but the ontology goes quite deep and can be used at multiple levels of detail.
The calculations are not difficult, I've implemented them twice in proofs-of-concept, including one that produces pretty charts.
The difficult part, to be honest, is that developing good estimates is difficult and frequently uncomfortable and the gains are not easily internalised.
Additionally, serious tool support is lacking in the places where it would make a difference -- issue trackers, for example.
[0] https://www.amazon.com/Measuring-Managing-Information-Risk-A...
edit -- Another good book in this area is Waltzing with Bears by DeMarco & Lister. A short, funny, insightful read, as you'd expect from the authors of PeopleWare: https://www.amazon.com/Waltzing-Bears-Managing-Software-Proj...
a) everyone else needs to do this across the board
b) it's still just a guess - normalising my guess and you're guess is hard
True, but that's where the calibrated probability assessment stuff comes in. If you can at least establish obviously correct (if very course grained) upper and lower bounds for estimates and then gradually shrink them in until you aren't confident doing so anymore, you have something at least slightly better than a complete guess. And if you can choose the correct distribution that the actual values would vary over, then you can use Hubbard's approach using a Monte Carlo simulation, and get some insight into likely outcomes.
No, it's not a perfect approach, but it gives you a little something to hang your hat on.
It’s really not very good, even for executives who shouldn’t care for technicalities. The best thing are the calibration exercises. But my advice is, skip this one.
I'm half-way through it. I know most of the general stuff already but my knowledge is from lots of sources I mostly forgot. This books seems to be a good collection for this topic. At least, I don't know any substitute.
For example: it tells you to do Monte Carlo simulations with made-up probability distributions, but silently lets the issue of the joint distribution -- how the made-up random variables correlate. But so much risk is driven not by marginals (say, I know the physical parameters for my chair and have a certain confidence that it won't crumble like carboard) but by correlations, even apparently distant ones (there's fracking or whatever going on and destabilizing the upper layers of the Earth, increasing the chances of earthquakes).
An even broader critique is the McNamara fallacy:
> "The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide."
----
There's this book I like called "Guesstimations in the back of a napkin" or something -- it's a book of exercises in Fermi-type estimation. It encourages you to consider the whole, to think bottom-up, top-down and core-out. It preserves a keen sense of the qualitative complexity of problems even as it encourages you to, well, make wild guesses.
This? https://www.amazon.com/Guesstimation-Solving-Worlds-Problems...
Compared to what?
But my advice is, skip this one.
And read what instead?
Not trying to start an argument here, I'm genuinely curious, as I consider How To Measure Anything to be one of the best books I've ever read (and I read a lot of books), and I recommend it highly to, well, pretty much everybody. If you feel that there's a better resource out there that relates to these topics, I'd be curious to know about it.
Exercise books for Fermi estimates like Guesstimation, etc.
Further out, something in systems thinking, maybe Donella Meadows' "Thinking in systems". Further further out, maybe those Stafford Beer papers about the Viable Systems Model? At one point Beer and Allende thought they were about to implement Red Plenty.
---
I understand the businessy logic that nothing is so fundamentally qualitative that it shouldn't be quantified. But you'll always be safer if you keep rich qualitative models and treat quantification as gravy on top of that.
The extreme opposite of rich qualitative models is the Soviet method of material balances. Halfway through there's the McNamara Fallacy:
Yeah, Fooled by Randomness and The Black Swan were both pretty good. I haven't necessarily thought of them as significantly overlapping with the HTMA stuff up until this point, but now that you mention it I can see a connection. I should probably go back and re-read both, and read Antifragile.
maybe those Stafford Beer papers about the Viable Systems Model?
Hmm... never heard of "Viable Systems Model" before, so I'll have to go read up on that. Thanks for the pointer.
Exercise books for Fermi estimates like Guesstimation, etc
I'll take a look at Guesstimation. Thanks for the pointer on that as well.
But you'll always be safer if you keep rich qualitative models and treat quantification as gravy on top of that.
I can buy that. I'm a fan of using approaches like Hubbard's to quantify things to a point. I do think his approach can supply a bit of extra rigor and some useful bounds to things that otherwise seem impossible to quantify at all. But it's not a perfect system by any means. The two biggest risks, so far as I can tell, would be leaving a variable (or more than one) out of your model completely, or using the wrong probability distributions for the various variables when doing the simulation part.
Netflix is just going straight to hell with such prediction approaches. And, mind that, this is not a prediction, this is a fragility statement.
At what point does the tail represent "all humanity dead"? No one needs a contract for the humanly unrecoverable.
Some event may cause an economic crash, companies can have insurance against business disruption. If an event suddenly causes a lot of insured business disruption, that might mean a large total payout.
Also not clear why loss frequency is assumed to be independent from loss magnitude.
Instead of "percent chance that the loss will exceed X" (VaR), expected shortfall would be more useful.
LEF and LEM are built of lower factors that roll up, which to my mind at least seem independent.
I'm not sure what you meant by expected shortfall.
I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.
Read Black Swan, by NNT. His thesis is that the most significant impacts on an individual and/or a collective are the function of outliers or "Black Swans". In a Malcolm Gladwell kind of way, he uses a bunch of anecdata to prove his thesis. It's a pretty good book I suppose, but it was also before the all the faux-philosophy Malcolm Gladwell type books became a thing.
>I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.
What exactly do you mean by that? In all sincerity, I think you misused an idiom or at least didn't make a clear connection as to why you chose that one.
In my opinion this is perhaps the best 1 line description of his work in general. I think there are some interesting ideas which are worth discussing, but sometimes his attitude and use of "anecdata" tires me out.
I think it's also worth saying that when people say reputational risk they're generally not thinking about litigation, they're thinking about consumer perception, which it's not crazy to believe would have large impacts for large companies.
IME, the biggest problem in this space is: on the one side, you have all kinds of metrics related to hardware, software and operations in your company; on the other side, you have FAIR and related ideas that let you model scenarios in a sane way, and let you plug in those statistical methods. But, there's no good idea how to connect these two sides. There is a vast gulf between what you can reliably and accurately measure, and what you actually want to know. It's hard to cross it while maintaining any kind of accuracy or statistical validity. I've been doing work on this "middle layer", but never got the chance to test it out in the open, as the company I worked for suddenly imploded.
(If anyone is working in this space and is accepting remote workers, I'm available for a chat.)
I'd be happy to discuss this topic with anyone; my email address is in my profile.
I click the link.
Sniff/Accountwall stops me. I hit ctrl+w before I even thought of doing it (yes for muscle memory!)
For people who went through the trouble to see the content, is the library worth it?
These days, anything behind paywall/sniffwall/must-be-authenticated-to-read-this-wall warrants a ctr + w from me.
Your reply could have quoted my reply, with a comment that said "maybe, but [insert counter argument, or poke holes in my statement's logic]".
Tab closed.