Bandit algorithms can also be approached from a Bayesian point of view! This lets you quantify the uncertainty in your estimates (e.g. how uncertain you are than ad A has a higher CTR than ad B), which a lot of other bandit methods don't offer.

To toot my own horn a bit, I wrote a blog post about Bayesian bandits: https://eigenfoo.xyz/bayesian-bandits/