Hacker News Reader

Comment by GistNoesis | Hacker News Reader

The "trick" is that word vectors are just projections of sparse tensor.

For example the word "queen" can be represented by its attributes. Representing in triplet form from wikidata Q16511993 (queen, subclass ,monarch),(queen, subclass, female),... A king would be (king, subclass ,monarch),(king, subclass, male),

Transitioning from queen to king is just changing an attribute in this triplet space, i.e removing a triplet and adding another. All those operations are linear. And then you take the projections.

This is kind of similar to the kernel trick of SVMs. You do the calculations in a lower dimensional space (word vectors) but implicitly you are doing them in the higher dimensional space (graphs of relations, aka triplets lists).

View on HN

One contention I have with this description (if I understand it at all), is that the projection into the linear space does not preserve the properties of changes in a relation graph.

I mean I "think" I said pretty much the same in my reply, but one big issue with word2vec is that the relations, which are graphical, can be projected, but not all changes in the graph can be. Therefore, I think it is not correct that these operations are linear in the sense that they can be equivalent to linear operations in the embedding space.

But maybe I just don't understand your description.

The sparse tensor perspective intrigues me, could you perhaps provide me with some references?

that person's description is unnecessarily jargony. sparse tensor just means vector that's mostly zero and has some ones. take your corpus of concepts and number them 0 to n-1. then each word you want to turn into a vec turns into a vector that has ones for concepts it's associated with and 0 elsewhere. learning that association is what word2vec does.

Can you explain attention in this same way?

nah i don't know much about LSTM but this looks like a fine tutorial

https://towardsdatascience.com/intuitive-understanding-of-at...

Not specifically. You can look at the literature for ontology, relation networks, semantic databases, reasoning.

The video that made me clicked, was https://www.ted.com/talks/ajit_narayanan_a_word_game_to_comm... It depicts in a game to help autistic children better communicate, a simple order-independent language representation, which is isomorphic to graphs.

If you do the math, you see that some triplets query like (queen, subclass, ?) is just a point-wise multiplication with the knowledge database sparse tensor in this sparse boolean tensor representation (the ? is a column of 1). Then it's just chaining requests.

A graph of triplet is isomorphic to a boolean tensor of dimension (nbSubjects, nbRelations, nbObjects).

Reply on news.ycombinator.com