Text embeddings reveal almost as much as text

One classic old estimate of the amount of information per English word, from Claude Shannon, is 11.82 bits per word. So very roughly, a 32-token text – the length they highlight to demonstrate their technique – should be representable in about 379 bits.

One of the embeddings they demonstrate the use of their technique against is the `text-embedding-ada-002` OpenAI offering, which gives back a 1,536-dimension representation, where every dimension is a floating-point number.

So as a matter of theory, just in the sign-bits of those vectors, there's enough raw state to potentially encode 32-word texts, and more.

If those float-dimensions are 4-byte floats, as are common, a single `text-embedding-ada-002` text vector takes (1536 * 4 bytes =) 6,144 bytes of storage (49,152 bits), While the dense/continuous nature of these values, and all the desirable constraints/uses packed into them, means you won't be getting that much precise/lossless text-representation from the values, it's plenty spacious for capturing a pretty-good compression of some pretty-long texts.

The interesting thing here is how often that short texts can be perfectly or nearly-perfectly recovered, via the authors' iterative method – even without that being an intended designed-in capability of the text embedding.

You could also see the result as an example of the old saw, "compression is intelligence" (or vice-versa) – an insight which has motivated, among other things, the Hutter Prize.

Some interesting questions for future work on the same codebase could be:

• at what text length does accuracy peak, and at what length does it precipitously decline?

• are even the failures still reasonable summaries of the original text?

• if for some applications such recovery is undesirable – in many it's no problem – are there ways to change the models, with different/extra training/ablation/whatever, that retains other aspects of the embeddings' usefulness but impairs verbatim text recovery?

First author here! I thought it was interesting that so many people thought this was obvious after-the-fact. What we're trying to convey is that although inversion turns out to be possible, it's not trivial.

Here are the three levels of inversion model described in the paper:

(1) Training a decoder to invert embeddings gives you some similar text, but only with some overlapping words.

(2) Training an encoder-decoder with the embedding projected up to a 'sequence' for the encoder works better, and gets more words back, but almost never gets the right text back exactly, especially when sequences are long.

(3) Training this fancy architecture as a 'correction' model that recursively corrects and re-embeds text works really well. It turns out that search is necessary for this kind of thing, and can get many texts back exactly.

A fourth option would be to simply scale the inversion model up by 10x or 100x, which would give us predictably better performance. We didn't try this because it's just not that interesting.

Why wouldn't they? If you think of embeddings as a learned hash, and your hash space is wide enough, the embedding would simply be another, lossless representation of the input. The challenge, of course, is that inverting hashes is difficult. Except in machine learning, the hashes are typically intended not to be so; to preserve semantic and syntactic relationships, as word2vec famously demonstrated. And there are even text embeddings that use sub-word information like character n-grams, which can trivially represent rare words, like the kind embodied by personal information.

edit: Given the author agrees, I suppose the research question is how well and cheaply you can do it, across different embedding algorithms. For practitioners, the lesson is to treat embeddings like personally identifiable information, as the authors state in the conclusion.

Recently I was quite surprised how good inversion from the OpenAI ada-002 embeddings can be: https://twitter.com/thesephist/status/1698095739899974031

Diffusion based image generation is a kind of ‘reversing the embedding’, right?

The iterative error correction applied here almost feels like repeated noise reduction, too.

So does this mean you could use this sort of text recovery to do the same kinds of things image diffusers do, traversing the latent space and recovering images along a path - but instead recovering text that morphs?

Could we finally use this to discover the text that lies exactly halfway between the Declaration of Independence and the lyrics of ‘Baby Got Back’?

I guess not until it scales to more than 32 tokens. But we can dream.

If you want to play with a demo of this kind of thing I suggest this Colab notebook: https://linus.zone/contra-colab - via https://twitter.com/thesephist/status/1711597804974739530

It demonstrates a neat model that's specifically designed to let you embed text, manipulate the embeddings (combine them, average them or whatever) and then turn them back into text again. It's fascinating.

So like an autoencoder for text?

During the Word2Vec era, I used to average a few word embeddings to get centroid embeddings. My observation was that the average embed was still close to all the original embeddings up to 5 concepts. I tested with similarity search. Can't pack too many distinct meanings into a single embed, but you can pack a few.

My only gripe with word embeds was that they were mixing synonymy and relatedness. Even worse, mixing up synonymy with antonymy - hot and cold are similar in a way, but also completely opposite.

What is a text embedding?

Does this have any implications for training models of copyrighted material?

I’m sorry but the abstract doesn’t say it: What is a text embedding?

Does this mean you can store somewhat lossy text/data only as embeddings? Or is that generally a not so good idea...