Embedding Archives: Millions of Wikipedia Article Embeddings in Many Languages | Hacker News Reader

Embedding Archives: Millions of Wikipedia Article Embeddings in Many Languages

txt.cohere.com

75 pointszerojames9 comments

Seems like it would lock in a user to cohere proprietary model in order to embed new search terms to the same space?

Would love to see similar but embedded with a more open representation model or even sent2vec

Unfortunately yes, as the corresponding model is not also open-source.

There's a lot you can do with the vectors themselves without needing to embed any more text (e.g., clustering, exploration, visualization after dimensionality reduction...etc). Here's a previous embeddings exploration of top HN posts: https://txt.cohere.com/combing-for-insight-in-10-000-hacker-... A lot of that code can be used here as well.

If you want to query for a search term, you can use a trial API key which is free to use for prototyping. The embedding model itself is not open source, though. [co-author of the post here]

If that's the intent, IMO the release dataset should have more metadata (e.g. paragraph heading, article taxonomy)

How would you add that data? As new columns you mean? Or add the paragraph headings to the text of the paragraphs before embedding them?

New columns.

For the headings, I mean the Wikipedia section headings (which isn't always a paragraphs, my mistake).

In both cases the data can be used like to classify/visualize Show HNs in your linked post.

In a few of the examples given, WikiData would also be searchable enough to find the relevant articles to pull in. I've actually been really curious if you could use recent advancements in ML to augment the filling out of WikiData, since it's a bit sparse as is.

How recent is the Wikipedia snapshot used?

Edit, going to [1] the datasets are labelled 2022-12.

[1] https://huggingface.co/Cohere

If you want something similar to this that is fully downloadable and open source, check this out - https://huggingface.co/NeuML/txtai-wikipedia

It's an embeddings database of Wikipedia abstracts with page view data integrated to enable filtering pages based on popularity in addition to similarity.

Reply on news.ycombinator.com