What are embeddings? | Hacker News Reader

What are embeddings?

vickiboykis.com

257 pointsAnon8451 comments

An array of floats (an n-dimensional vector) which represents some piece of data like a text or an image. Different embeddings can be more or less close to each other, and this closeness indicates similarity.

^^^ this is a really succinct explanation. I would just add that an embedding is an array/vector of numbers. Does not have to be floats although they are in practice.

The magic is creating the embedding, embeddings themselves are nothing special.

Yeah, I'd been working with them for ages before I really wrapped my head around the sausage machine. There was a Wolfram blog a while back that really helped me grok it [0]. The last state of a neural network prior to prediction, if I understand correctly.

[0] https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

Not just similarity. It also includes all other possible relationships if you want, such as order, connectivity, etc.

What is order/connectivity in R^n?

It's pretty arbitrary. For example, you can define order as the product of all order relationships along each single dimension (binary product). As for connectivity, they can also be defined as boolean functions under certain radius, like the persistent homology spectrum.

I'd say similarity, but any and all kinds of similarity you could ever think of - as long as you have enough dimensions and training data to feed it.

My current belief is that a hundred thousand+ dimensional latent space, as in SOTA LLMs, seems to have enough dimensions to reduce a large array of cognitive skills into a proximity search.

Generally, it is any transformation that preserves relevant structure.

Importantly it’s a vector space with a metric. Not just an array of vectors. The original Word2Vec paper had the jaw dropping “king - man + woman -> queen” which needs the ability to do addition/subtraction on vectors. Also words “close” in meaning result in vectors “close” in some sense e.g. it needs a formal measure of distance (eg cosine similarity) which you hint at.

Paper: https://github.com/veekaybee/what_are_embeddings/blob/main/e...

I'd like to find a way to start with an embedding and have the computer generate some text that corresponds, at least approximately. There are tools that do that for images, right? Like Stable Diffusion, you can put an image in, get an embedding, then do gradient descent in latent space to find a new embedding, then generate a new image from that.

The most basic of embedding (that I can think of) is one where the number of dimensions corresponds to the number of unique characters in your lexicon. If there is a number in one of the components of your embedding that is greater than "0", then you know what characters they are. This embedding does not encode the order of the characters, though. They are just a "bag of characters". If you were to then also encode the order of the characters in, say, yet another embedding, you could use those two embeddings to recreate the original word.

Combine the two embeddings into a new vector space and BAM, you've invented "embedding2word".

Gpt (and many others) just add these embeddings together in the model, so you could do that and have one vector that encodes both things together

That seems suboptimal, for the same reason LLMs are trained on tokens, and not characters. Tokens seem like a much better "unit of meaning" than characters.

Curiously, this also applies to humans: we learn words first, spelling later; we almost always think in terms of words, subwords and phrases, and we're really fast at it - while anything to do with spelling seems to demand much more focus, and some degree of conscious hand-holding.

As for encoding sequence, I'm curious how that happens and need to find relevant papers, but I imagine there might be some "n-gram dimensions" in the vector, as in one value for "I'm a first token in a sequence", one value for "I'm a second token in a sequence", etc., which would encode the "occurs before"/"occurs after" relationships using few dimensions, leaving the rest for more interesting relationships.

You can get this fairly trivially with word and sentence embeddings just by running the inverse (huggingface models have this as a builtin). For llama, the same is possible, but the matrix transpose is your responsibility :)

Ok, I got annoyed at oobabooga (around when it first came out) and have been messing with llama.cpp since then. I can't tell if this feature request is the same thing we're talking about here? https://github.com/ggerganov/llama.cpp/issues/1552 I guess if I want more features, I should move on to something that has more features lol.

Looking at that issue, I wonder how is it that everyone seemed to not understand the poster's question?

The feature itself is something I wanted to play with too, as it's kind of an obvious thing to want. I mean, these models execute a pipeline:

[text] -> [tokens] -> {[embeddings] -> [inference] -> [embeddings]} -> [tokens] -> [text]

Where the part in { ... } may or may not be implemented as a single step (i.e. all three parts interleaved).

Now, apparently all the magic[0] of transformer models sits in the latent space and is invoked by the { ... } bit. We also know for sure that you can make the pipeline look like this:

[text] -> [tokens] -> {[embeddings] -> [inference]} -> [embeddings]

So with the two things in mind, it's kind of obvious you'd also want a pipe that looks like:

[embeddings] -> [inference] -> [embeddings] (and optionally -> [tokens] -> [text])

for the sole purpose of messing around and exploring the latent space itself.

I'm very much not up to date with the whole space, so I might be missing something, but I'd thought that poking around the latent space would be getting a lot more attention than it seems to be getting.

(EDIT: replaced < ... > with { ... } for readability.)

[0] Not the "how do transformers tick" details, but the "how the hell are they this good" / "GPT-4 is uncanny valley" / "could this thing be actually thinking?" kind of magic.

In every model I’ve worked with, all of those steps are EXPLICIT. You take input, then tokenize, then embed, feed to model, then convert from embeddings to tokens, then back to words. There’s actually a lot of work done in embedding space! Chained execution, neural graph reasoning (see https://arxiv.org/pdf/2303.14617.pdf), even basic IR tasks like clustering and retrieval.

There’s been no real effort to especially expose it because that is what you get by default. Even OpenAI has an “embed” endpoint. So you’re not going to see a huge push for it the same way you won’t see a push for reasoning about websites “in the HTML”. :)

I only see a way to get embeddings out. How do I put embeddings into an OpenAI API?

im not sure i follow if you want embeddings for any word/sentence you send it to the embeddingendpoint and save the vectors coming back?

The point is that we want to shove arbitrary embeddings vector into an LLM and inspect what comes out. This is different from feeding tokens as input. The kind of thing I'd like to do, for example, is to:

- Get embeddings for e.g. "blue" and "red", or "the sky is blue" and "galaxy redshift";

- Average them, resulting a vector that's bound to not be expressible with tokens alone;

- Input that to the same model I got the embeddings from, and see what comes out.

If by "embeddingendpoint" you mean OpenAI, they provide that for a specialized model, and (AFAIK) you can only get embeddings out (for the purpose of comparing various vectors yourself). They have no API endpoint for an LLM that can take those embeddings as input.

Run the resultant embeddings through local beamsearch to get tokens, then run the detokenizer to get back words.

You can't and you can only get embeddings from a tiny gpt-3 model.

This is basically how RNN encoder/decoder architectures worked. The encoder encoded the input as a single vector and the decoder would decode this into text (eg. for machine translation) [1]. However, fixed-length vectors generally required too much 'compression' to represent variable-length text, so people started adding attention mechanisms so that the decoder could also attend to the input text. And the seminal Transformer paper by Vaswani and others showed that you only need an attention mechanism and you could ditch the RNN (hence the title 'Attention is all you need') and here we are.

So, this has been possible already for quite a long time.

[1] https://arxiv.org/pdf/1409.3215.pdf

To be sure, the seq2seq style already allowed multi length embeddings before transformers were a thing. RNN vs Transformer is entirely an implementation choice; you can build a seq2seq model with any combination of transformer, RNN, and conventional layers.

Each of these layer types have different computational costs for training and inference, and encode different inductive biases, which may be more or less appropriate to a given problem.

For some models, perhaps you can use get_output_embeddings() like in this example: https://stackoverflow.com/a/65267429

Something that is confusing to me is how to use embeddings outside of trivial use cases. Say you want to build something that takes a natural language query, then uses a heterogeneous data set with, say, three different types of data, such as text documents, database schema, and single phrase tags, and you want to create an embedding that maps that native language request to a particular table in a particular database. I have no idea where to begin.

Shameless plug, I just wrote about Vector Databases and Embeddings 2 weeks ago in my newsletter (5 minute dev trends):

https://unzip.dev/0x014-vector-databases/

It also explains how to use them IRL and why.

thank you!

I'm glad you like it.

Nice post, thanks for sharing! Learnt about new dbs / solutions for vector search from it.

Vicki mentions this survey I wrote some time ago: https://towardsdatascience.com/milvus-pinecone-vespa-weaviat...

Hoping it'll be useful as well!

OP you need to make it clear that it's a book. The website is confusing.

There is a big green button that says "Get PDF" when I visit it.

iirc HN has a rule/guideline to add [pdf] to the post title in that case, although it might not always be enforced

My mind automatically dismissed that in the same way it dismisses site headers, ads, mailing list sign up links etc.

But even if I did consider it, a link called Get PDF wouldn't make me think it's a book. PDFs can contain books but they usually don't.

This looks really good and will add to my reading list.

Bear with me on this question as I am still learning...

From a quick skim I couldn't find it. It might be there? But with a transformer, at which layer do you consider the vector to be the embedding? Or are there multiple choices?

For example you have a positional and token embedding - so they are of course embeddings, but simple ones. King - Man + Woman = Queen type thing.

But I imagine you might want to consider the output of the 1st ? 2nd? Nth? transformer on the decoder side as an embedding of sorts.

For example the sentence "attention is all you" has gone through a couple of layers you now have a vector that understands those words pretty well. But if you go too far through I imagine you get closer to the "prediction" and so lose the meaning and just get something closer to "need".

But if possible to get, that sentence embedding (by which I mean N tokens up to context size) might be useful for search indexing of long text.

There are multiple choices. For example BERT authors use the pooled output from the last four layers after experimenting with a few other choices like concatenation and only using the last or penultimate layers.

Half way through and though I’m enjoying it, it’s not super clear at times.

For example: the Flutter example is introduced, then not mentioned for a while, then mentioned again with a specific problem, then ignored again, then used again with some other different problems. I keep getting lost about what we’re actually trying to achieve with it, and sometimes it feels like it’s entirely unnecessary to refer to it.

There are other times where we jump from pretty simple explanations to an intense matrix decomposition. A lot of jargon also gets used that, though previously explained, is used really densely. I feel like keeping the wording simpler would still get the point across.

Other than that, I’m finding it quite useful, and I’m only on page 30/75. Thank you for writing it :)

(my experience: CS degree, self-studied some AI stuff for the past half a year)

A mapping whose codomain is an N dimensional space.

Don't know why you're downvoted since you're absolutely correct.

They're just locality-sensitive hash functions.

Those seem like two very different definitions.

    lsh: similarity(x, y) < thresh => E[P(d(lsh(x), lsh(y)) < 1)] > 1-eps for some eps

For a model that learns a good representation, and some suitable distance function to measure distances between embeddings (e.g. Euclidean)

    model: similarity(x, y) < thresh => E[P(d(emb(x), emb(y)) < r] > 1-eps

for some cutoff distance r in the vector space

The difference in definitions is that my definition does not imply that the mapping maps similar objects to similar locations in space. Embeddings that do can be useful which is why you see people using machine learning to create them, but it is not a necessary property of an embedding.

True, it’s not a necessity just based on the types of things, but in practice it’s the intended use

I'd imagine it's because it's the title of the thing being linked to, not an actual question to be answered.

Farting out a quick one line answer is a boring comment

There is "wikipedia article summary" kind of correct, but that doesn't mean resources that are aimed at teaching are not valuable too.

> Don't know why you're downvoted since you're absolutely correct.

OP might be down voted because, even though the answer might be technically correct, it still completely fails to explain the concept (i.e., it's a useless explanation) and does so with a hint of arrogance and demeaning to readers.

Yours too, by the way.

It's like someone asks what is temperature and some smartass answers it's a scalar quantity that's mapped from a physical state into a semi-infinite continuous set. Great to know.

A.k.a. a matrix.

This is the kind of explanation where, by the time you can understand it, you don't need it. Except as a reminder I guess. But it's useless to beginner and intermediate learners.

I think there's value in that style of definition.

I found "a monad is just a monoid in the category of endofunctors" the most useful explanation of monads. I didn't know what an endofunctor—or a moniod—was when I read it, but it gave me some hooks for research. Other explanations using analogies just left me confused, and nothing further I could look into.

Though that said, I revised that explanation after the others did not help me, so maybe I had absorbed enough context.

The way that this book answered its title question was unsatisfying to me. It felt like it was so focused on the data structure of embeddings that it totally glossed over questions about what the structures mean, why they work so well, and emerging research issues in embeddings.

It's like if someone wrote a book called "What is consciousness?" and just provided an annotated list of neurotransmitters. That approach leaves a lot out.

Reply on news.ycombinator.com