LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

Theoretical efficacy doesn’t guarantee real world efficacy.

I accept that this is self reinforcing but I favor real gains today over potentially larger gains in a potentially achievable future.

I also think we are learning practical lessons on the periphery of any application of AI that will apply if a mold-breaking solution becomes compelling.

Two exmaples. (1) grammars in natural languages. You can just see in another commenter here uses "a local maxima", and then how people react to that. I didn't even notice becuase English grammar has never been native to me. (2) Mostly, prepositions between two languages, no matter how close they are, don't have a direct mapping. The learner just has to remember it.

I know some of the smaller models like PHI-2 are training for reasoning specifically before by training on question answer sets, though this seems like the opposite to me.

I think of how tSNE and other algorithms sometimes produce projections that sometimes look like that (maybe that’s just what you get when you have to bend something complicated to fit into a 2-d space) and frequently show cusps that to me look like a sign of trouble (took me a while in my PhD work to realize how Poincaré sections from 4 or 6 dimensions can look messed up when a part of the energy surface tilts perpendicularly to the projection surface.)

I still find it hard to believe that dense vectors are the right way to deal with text despite the fact that they work so well. For images it is one thing because changing one pixel a little doesn’t change the meaning of an image, but changing a single character of a text can completely change the meaning of the text. Also there’s the reality that if you randomly stick together tokens you get something meaningless, so it seems almost all of the representation space covers ill formed texts and only a low dimensional manifold holds the well formed texts. Now the decision surfaces really have to be nonlinear and crumpled over all but I think there’s a definitely a limit on how crumpled those surfaces can be.

I think a more interesting question is how much information there is on the internet, especially after optimal compression. I'm guessing this is a very difficult question to answer, but also much higher than LLMs currently store.

> Finally, we use our dataset and LRE-estimating method to build a visualization tool we call an attribute lens. Instead of showing the next token distribution like Logit Lens (nostalgebraist, 2020) the attribute lens shows the object-token distribution at each layer for a given relation. This lets us visualize where and when the LM finishes retrieving knowledge about a specific relation, and can reveal the presence of knowledge about attributes even when that knowledge does not reach the output.

They're just looking at what lights up in the embedding when they feed something in, and whatever lights up is "knowing" about that topic. The function is an approximation they added on top of the model. It's important to not conflate this with the actual weights of the model.

You can't separate the hallucinations from the model -- they exist precisely because of the lossy compression.

I have the same question.

Peter Norvig’s GOFAI Shakespeare generator example[1] (which is not an LLM) gets impressive results with little input data to go on. Does the leap to LLM preclude that kind of small input approach?

[1] link should be here because I assumed as I wrote the above that I would just turn it up with a quick google. Alas t’was not to be. Take my word for it, somewhere on t’internet is an excellent write up by Peter Norvig on LLM vs GOFAI (good old fashioned artificial intelligence)

Modern LLMs are able to transfer knowledge between different languages, so it's fair to assume that some mapping between human language and a more abstract internal representation happens at the input and output, instead of the model "operating" on English or Chinese or whatever language you talk with it. And once this exists, an internal "world model" (as in: a collection of facts and implications) isn't far, and seems to indeed be something most LLMs do. The reasoning on top of that world model is still very spotty though

I think it's more likely that LLMs are actually learning and understanding concepts as well as memorizing useful facts, than that LLMs have discovered a compression method with that high of a compression ratio, haha.

> Is that a good summary?

No - there's a lot more going on. It's not just mapping input patterns to output patterns.

A good starting point to understand it are linguist's sentence-structure trees (and these were the inspiration for the "transformer" design of these LLMs).

https://www.nltk.org/book/ch08.html

Note how there are multiple levels of nodes/branches to these trees, from the top node representing the sentence as a whole, to the words themselves which are all the way at the bottom.

An LLM like ChatGPT is made out of multiple layers (e.g. 96 layers for GPT-3) of transformer blocks, stacked on top of each other. When you feed an input sentence into an LLM, the sentence will first be turned into a sequence of token embeddings, then passed through each of these 96 layers in turn, each of which changes ("transforms") it a little bit, until it comes out the top of the stack as the predicted output sentence (or something that can be decoded into the output sentence). We only use the last word of the output sentence which is the "next word" it has predicted.

You can think of these 96 transformer layers as a bit like the levels in one of those linguistic sentence-structure trees. At the bottom level/layer are the words themselves, and at each successive higher level/layer are higher-and-higher level representations of the sentence structure.

In order to understand this a little better, you need to understand what these token "embeddings" are, which is the form in which the sentence is passed through, and transformed by, these stacked transformer layers.

To keep it simple, think of a token as a word, and say the model has a vocabulary of 32,000 words. You might perhaps expect that each word is represented by a number in the range 1-32000, but that is not the way it works! Instead, each word is mapped (aka "embedded") to a point in a high dimensional space (e.g. 4096-D for LLaMA 7B), meaning that it is represented by a vector of 4096 numbers (cf a point in 3-D space represented as (x,y,z)).

These 4096 element "embeddings" are what actually pass thru the LLM and get transformed by it. Having so many dimensions gives the LLM a huge space in which it can represent a very rich variety of concepts, not just words. At the first layer of the transformer stack these embeddings do just represent words, the same as the nodes do at the bottom layer of the sentence-structure tree, but more information is gradually added to the embeddings by each layer, augmenting and transforming what they mean. For example, maybe the first transformer layer adds "part of speech" information so that each embedded word is now also tagged as a noun or verb, etc. At the next layer up, the words comprising a noun phase or verb phrase may get additionally tagged as such, and so-on as each transformer layer adds more information.

This just gives a flavor of what is happening, but basically by the time the sentence has reached the top layer of the transformer it has been able to see the entire tree structure of the sentence, and only then have "understand" it well enough to predict a grammatically and semantically "correct" continuation from which it is able to predict continuation words.