Hacker News Reader

Do you have a good textbook to reference to where the theory starts from PAC-Bayes?

You can try this one

https://books.google.co.in/books/about/User_friendly_Introdu...

Free download here

One of the ICMLs had a nice tutorial by Langford and Banerjee on the relationship between the different style of bounds. 2003, I think.

Just to add on top of the quality reference provided by srean, I like to first drill in Bayesian principles and then use this article to derive PAC-Bayes from that: https://arxiv.org/abs/1605.08636

Regular PAC falls out by taking a uniform prior over a finite hypothesis class (and then building up VC dimension if desired, but usually by this point you realise why the bounds are unlikely to be good).

Seems I was misremembering the dates. The Langford and Banerjee papers/turorials I had in mind were

On Bayesian Bounds https://dl.acm.org/doi/10.1145/1143844.1143855

Tutorial on Practical Prediction Theory for Classification https://jmlr.csail.mit.edu/papers/v6/langford05a.html

The first one is quite in the same spirit that you like.

Reply on news.ycombinator.com