Are you looking for books that teaches advanced graphics? Some topics I think some topics along that line are perlin noise, parametric modeling, computational geometry, and math for game developers. A lot of advanced graphics stuff can be seen in WebGL and Three.js, but starting by googling “graduate computer graphics course” can get the gears running. I will post some links I find interesting.
Graphics: look at some of cornell’s course links: http://www.cs.cornell.edu/courses/cs4620/2015fa/index.shtml
List of graphics courses: http://www.cs.cornell.edu/~kb/
For VR, an amazing course taught by Steven Lavalle: http://vr.cs.uiuc.edu
For AR, it’s hard to dive into cause AR is heavily ingrained with Computer Vision. I’ll look for more and post here. Hope this helps!