I think this has something to do with how to brain stores memories, and that storage is highly entwined with a spatial location. Somehow being able to tie that memory to a location in space makes it more concrete for me. If I remember a fact from a book I can also usually remember the physical location of the fact inside the book.

My point is that it's not a conscious process, so the book/map isn't replaced by a simple location indicator. The experience of a book isn't reducible in this way.