Having played around with embeddings and built a few production use-cases with them.. They are awesome and enable a lot of cool applications. But, if you're building in a particular domain, you'll hit limits with any off-the-shelf embeddings model. One way to think about this is that the off-the-shelf embedding models have a lot of dimensions, but some of those dimensions may and some may not that matter for your application (classification, content similarity, clustering, etc). In other words, a vector may be close to another vector in dimensions you don't care about.

I look forward to seeing better tooling and literature around fine-tuning embedding models.