Could you briefly sketch out the math (or point to a sketch) so that other people can pick it up? I'm quite interested!
(I suspect you could get pretty far with a Monte Carlo simulation, and that would let you bypass most of the math anyway.)
(I suspect you could get pretty far with a Monte Carlo simulation, and that would let you bypass most of the math anyway.)
Here's a nice presentation on an analysis of the method, references start on slide 47
https://people.orie.cornell.edu/pfrazier/Presentations/2014....
Paraphrasing the theorem from Jedynak, Frazier, Sznitman (2011):
Suppose [flake probability] is constant, known, and bounded away from 1/2, and we use the entropy loss function. ... The policy that chooses [next test point] at the median of [current posterior distribution] is optimal.
And some python code that implements a version of the analysis
https://github.com/choderalab/thresholds/blob/master/thresho...
https://github.com/Ealdwulf/BBChop/blob/master/BBChop/doc/Ba...
(Edit- had wrong link originally)
When gathering more evidence, you'd use your new belief about which cookie bowl Fred has as P(H1)=0.6 and P(H2)=0.4
I was specifically interested in the application to binary search / bisection in the presence of flaky tests.