Yep and embeddings can be decoded back into meaningful text if you have the model weights and the tokenizer.