Postgres can do a lot of things, but for large enough datasets and/or when you want to add filtering into the mix along with vector search, then it becomes slow. And at that point you want to use a dedicated vector search database.
It's similar to how Postgres can also do full text search, but for large datasets and/or you want to add typo tolerance, faceting, grouping, filtering, synonyms, etc - the usual features you'd need when implementing a search feature - then it becomes slow to do this in pg and you'd then use a dedicated search engine.
In Typesense, we've now combined Vector Search along with filtering based on attributes in your documents, so you get the best of both worlds [2], and we made it fast with an in-memory index.
[2] https://typesense.org/docs/0.24.0/api/vector-search.html