As a self contained, foundational course, Georgia Tech's OMSCS offering [1] is solid. Charles Isbell and Michael Littman are great at building intuition into equations.
[1] https://www.udacity.com/course/reinforcement-learning--ud600
[1] https://www.udacity.com/course/reinforcement-learning--ud600