Bertsekas' earlier books (Dynamic Programming and Optimal Control + Neurodynamic Programming w/ Tsitsiklis) are great references and collect many insights & results that you'd otherwise have to trawl the literature for.
I'm very interested to see what a book focused more narrowly on RL will be like-- Sutton's Introduction to Reinforcement Learning[0] is fantastic, but if you're going to do research on RL, another text such as this one is necessary.