Oh yeah! I wrote a similar algorithm and constantly suffered similar problems. Constraining the latent space can be a major pain. Its hard to visualize but I think a lot of latent features may exist in non-continuous latent "pockets", meaning that the desired direction of the latent vector is dependent on your position in space.