There is an excellent slide deck by Kayvon Fatahalian [0] that I consider to be the best high-level introduction into the topic (especially if you have a deeper understanding of how a CPU works). But I agree, more detailed insights would be great.

[0] http://s08.idav.ucdavis.edu/fatahalian-gpu-architecture.pdf

Here's a 100-page booklet on the hardware/software interface that covers a lot of that kind of stuff:

http://sbel.wisc.edu/Courses/ME964/Literature/primerHW-SWint...

The last 20 pages are on GPUs.

Disclaimer: I work in the GPU industry.

If you're interested in the architecture of a GPU this Berkeley ParLab presentation by Andy Glew from 2009 covers the basics of how the compute cores in modern GPUs handle threading. It's a subtle, but powerful, difference from SIMD or vector machines.

http://parlab.eecs.berkeley.edu/sites/all/parlab/files/20090...

If you want to get into the details of how a GPU interfaces with the system and OS software, which is almost an entirely other animal, you may want to look at the Nouveau project to get oriented.

http://nouveau.freedesktop.org/wiki/

The latest (5th) edition of "Computer Architecture: A Quantitative Approach" by Hennessy & Patterson covers GPU (among many other things). One chapter on data parallelism covers SIMD, vector processors and GPUs. Very good overall IMHO, but better check the content to see if it suits your expectations.