They should be quite similar. In the end you coax your embedding space to amount to some consistent measurement of what causes samples to diverge from one-another.

You can do similarity search and all the sorts of things you do for word embeddings on embeddings generated for other scenarios.