A single sentence can have a lot of different meanings. I wonder if it would be more fruitful to use rolling pairs and triplets? Ie was thinking about how you use two moving averages with different window sizes to detect trend shifts.
So we can see how as we are appending sentences and getting the embedding we get the movement of the story in the latent space.