Thanks for the response.

One thing to clarify: It's a 300K x 300K similarity matrix, which means I have 300K embeddings. Each embedding itself only has dimension 512.

In other words, the similarity matrix is the similarity between each embedding & every other embedding in the 300K set of embeddings.

Regardless, I think Dask will be useful here.