Nice writeup! Why do we love bandit problems? Not only the efficacy with which agents can minimize exploration and maximize rewards in unknown worlds. But also the emergent philosophical truths that arise concerning causality!

Remains very much an active research topic. With applications ranging from epidemiology, to website optimization ;)

CS7792 - Counterfactual Machine Learning, T. Joachims, Cornell University

http://www.cs.cornell.edu/courses/cs7792/2016fa/

Deep Bayesian Bandits Showdown - Google Brain

https://arxiv.org/pdf/1802.09127.pdf