> The difference in irreducible uncertainty between "shuffled cards" and "anything else" is a matter of degree, not kind.
Difference in degree or difference in kind is not the important point to sort out. The important question is what computation methods actually work in practice for humans. Some algorithms that work well on degree-small datasets fail to terminate in the course of the universe for large-datasets.
If you've ever operated in a particular uncertain environment (like, as a first time startup founder), it immediately becomes clear how totally useless explicit probability calculations are when making decisions. I was a self-proclaimed rationalist and tried reasonable hard to be a good Baysian; I ended up deciding I might as well sacrifice a goat and read it's entrails.
> The point isn't to follow the numbers off a cliff, but if you're well-calibrated (as in, have a track record with a decent brier score, or something), then pretending those numbers are totally useless is a bit silly.
I'm actually trying to make a stronger point that these numbers are "totally useless." I think in practice many applications of explicit probability calculation are actually quite harmful.
Rationalists love to talk about map and territory as the two items to be concerned about, but there's also another thing called "agent's belief in the effectiveness of their map." What models/explicit math/probability gets you in improved mapping, it takes away from you by making one's belief in the effectiveness of their map much much more.
If you look at large-scale model failure in practice, it's almost always because someone thought the math they were doing captured the full system in a complete way (and then it didn't). 2008 is a fantastic example of this.
> Well, you'd be able to do that in a world where we cared about keeping track of that sort of thing. Too bad people keep finding reasons not to do that, huh?
I'd love to start tracking predictions generally! I agree with you here!
> Surely you aren't telling me that you've updated your priors on observed evidence, and as a result have a different expectation about future world states (that is, "how useful would explicit Bayesian reasoning be if I tried using it")?
To be clear: it's the explicit probability calculation and mathification that I take my major issue with. I am most definitely not against learning from what I observe :-)
> Their level of performance is not compatible with the "the small number of people out millions for whom the the coin flip came up heads 10x in a row" hypothesis
Can you link this math? I'd love to see it - not flippant, genuinely looking to check it out and be educated here!