This sort of landscape memory has happened extensively with entirely unrealistic, fabricated worlds too, though (cf. MMORPGs (World of Warcraft) and even games such as DOOM).
Surely the fact that there are people out in the real world conversing with one another about places in your fabricated one is a selling point rather than something to be discouraged?