Grasp2Vec: Learning Object Representations from Self-Supervised Grasping

We were inspired by the (emergent) linearity property of Word2Vec representations (man - woman = king - queen). Vector addition (which is Abelian) make sense for representing sets of objects, e.g.

(cup, table, ball) = (cup, table) + (ball)

Unlike word2vec (where this property magically arises from training a language model), we enforce this property in our training objective in order to learn good visual models.

Unlike typical use of graph and relational networks, we explicitly wanted to throw away spatial relations between objects, to force the representation to keep track of what objects are still remaining even if the robot re-arranges the scene during the grasp. I think using a graph network makes sense if one wishes to model pairwise relations between objects though.

It begs the question of creating something like “2vec2vec”, a vectorization model that takes vectorization models as input and embeds them in a vector space that encodes representational semantics of vectorization models into linear operations in a vector space.

Of course then you just run 2vec2vec through itself and get the vectorized representation of a vectorized representation model of vectorized representations.