Skilling wrote a nice critique of information geometry that's worth reading:

http://djafari.free.fr/MaxEnt2014/papers/Tutorial4_paper.pdf

The main point is that KL-divergence is not a metric, so imagining it as a distance in a space may give you some wrong intuitions. Its matrix of 2nd derivatives, the Fischer Information, works as a local metric, but then many people want to draw global pictures that try to extend this back to a global metric, which doesn't actually work.