Gpt (and many others) just add these embeddings together in the model, so you could do that and have one vector that encodes both things together