We have created a binary embedding to help us with the embedding sizes. I think in the future we will see a lot more research in reducing the model and embedding sizes.

https://medium.com/ozonetel-ai/compressing-bert-sentence-emb...