Yeah, that's exactly right. You take the embedding of your query and then compare it to the embeddings of your documents (or chunks of documents).

You can use vector databases that implement the Approximate Nearest Neighbors (ANN) algorithm, or if you don't have too many documents you can just brute force the similarity comparison.

I'm not sure at what point you _really_ need or want a vector database, but anecdotally, calculating the Hamming distance for a couple thousand vectors seems pretty negligible.