Hacker News Reader

Comment by seydor | Hacker News Reader

This is confusing, using the semantic vectors arithmetic of embeddings is not very relevant to transformers and its completely missing the word 'attention'. I don't think transformers are that difficult to explain to people , but it is hard to explain "why" they work. But i think it's important for everyone to look under the hood and know that there are no demons underneath.

View on HN

I was trying to keep the article at a level that everyone understands, from middle school up. I thought about going a bit deeper in the structure and mentioning attention, but my problem is that the intuitive concept of "attention" is quite different from the mathematical reality of an attention layer, and I'm sure I would have lost quite a few people there. It's always a trade-off :)

Indeed the word 'attention' implies some active process, which is also unfortunate, because it is really similarity maps over long ranges. This process apparenty creates a cross-reference maps of items that are commonly co-activated. But then what do the different heads do? Some kind of different "modalities of cross-reference"?

I struggle to understand why this thing works the way it does. It's possible that Vaswani et al. have made one of the greatest discoveries of this century that solved the language problem in an unintuitive, and yet very unappreciated way. It's also possible that there are other architectures that can simulate the same level of intelligence with such large numbers of parameters.

I think you re right that it's not intuitive, it's like basic arithmetic is laughing at us

Multi-head attention just means that you're looking at all the words at once rather than only looking at one word at a time, and using that to generate the next word. So instead of using attention only on the last word you also have attention on the penultimate word and the one before that and the one before that, etc. I think it is fairly obvious why this gives better results than say an RNN – you are utilizing context better than in a recurrent system, which is also just closer to how a human brain works. When you read/write a sentence you're not really going one word at a time, you're thinking about all the words at once, even if the last word is technically the most important.

The other clear benefit of transformers over an arch like RNNs (and what has probably made more of a difference imo) is that its properly parallelizable, which means you can do huge training runs in a fraction of the time. RNNs might be able to get to a level of coherence that approaches GPT-3, but with current hardware that would be very time-prohibitive.

So in training, chatgpt turned words into embedding , and given context window N , looked at N embeddings an created a probabilities list for the following next embedding ?

And if I tell it something that was excatly in it's trained context windows, I get the most likely next word and the one after itm

But what happens if I ask it something slighty different than it's training context ? Or something largely different?

By "embedding" in this context what you're actually referring to is called a "token" which are sub-word strings of usually 1-4 characters.

It's not possible for you to ask it things even slightly different from it training data, unless you ask exclusively in emojis that didn't exist yet when it was trained (in which case it sees nothing, just like when someone sends you an emoji your phone doesn't support).

Any novel sentence and even novel words like "Blobdarfnk" ARE in its training data. "Blobdarfnk" is encoded as the five tokens Bl, ob, dar, fn, and k.

That's not what multi-head attention means. Multi-head attention is the use of learned projection operators to perform attention operations within multiple lower-dimensional subspaces of the network's embedding space, rather than a single attention operation in the full embedding space. E.g. projecting 10 512-D vectors into 80 64-D vectors, attending separately to the 8 sets of 10 embedding projections, then concatenating the results together to reform 10 512-D vector outputs.

In fact the projection operations are the only learned part of a Transformer's self-attention function -- the rest of self-attention is just a weighted sum of the input vectors, where the weights come from the (scaled) vector correlation matrix.

How is that different from what I said?

> I struggle to understand why this thing works the way it does.

I'm not in this field but have recently found myself going on the deepest dive possible into it as my small brain can absorb.

I now know about (on a surface level) neural networks, transformers, attention mechanisms, vectors, maticies, tokenization, loss functions and all sorts of other crazy stuff.

I come out of this realizing that there are some incredibly brilliant minds behind this. I knew AI was a complex subject but not on the level I've learned about now. To get what is essentially matrix multiplications to learn complex patterns and relationships in language is mind-blowing.

And it's creative. It can have a rap battle with an alter-ego, host a quiz party with other AIs of varying personalities, co-author a short story with me, respond to me only in emojis. The list is seemingly endless. Oh, and it can also do useful things. It's my programming companion too.

And we're just getting started.

Got a good Youtube list? Other than the HN threads and submissions I can look up this weekend.

this series is good https://www.youtube.com/watch?v=Nw_PJdmydZY

No worries, readers can just ask ChatGPT 'what's the meaning of attention in the context of LLM design' and go from there [edit also temperature].

> that there are no demons underneath

Given that Lacan already proposed the unconscious as structured language-like more than half a century ago and described attention in his turn on Freud's impulse in favor of his concept of derive, we may say, this is pretty much where our own demons live.

(I actually do think that revisiting Lacan in this context may be productive.)

We need to form some sort of guild of engineers who think Deleuze, Latour, Lacan et caterva should be read within our disciplines.

I second this. The STEM (= engineering) / humanities divide in software may miss some.

(There had been times, when linguistics were still a major entry path into computing, where things were a bit different. Notably, this were also the times, which gave rise to most of the general paradigms. A certain amount of generality was even regarded a prerequisite to programming. Particularly, HN is such a great place, because it holds up this notion of generality.)

Please no! Read systems neuroscience. Like Hassabis does. Or if of a philosophical persuasion, then Dennett or Rorty.

Much of cognitive science reinvents wheels that had been established in the 1920s and 1930s already, namely in sociology of knowledge and related fields. fRMI actually often confirms what had been already observed in a psychoanalytic context. (I don't think it's a good general advice to totally ignore what is already known.)

But Lacan? And no, there is a vast new world of cognitive neuroscience that was undreamed even 10 years ago.

> But Lacan?

Well, if you're in need of an established theory of (semantically driven) talking machines and what derives from this, and what this may mean for us in terms of freedom, look no further.

Can you recommend a specific work of his? What Lacan I have leaves me bemused by his brilliance but not informed. Dennett provides both without the fireworks.

Generally, don't start with the "ecrits" (writings), they are hermetic and you really have to have some head start on this. From the seminars, Livre XI, Les quatres concepts fondamentaux de le psychoanalyse (1964) may be a start, as it – in parts – aligns itself with the cybernetic research of the day. However, do not expect too much from a single reading or a single of the seminars.

(Mind that this is trying to talk about what's beyond/below language, necessarily using language just the same, which is – at least according to (the early) Wittgenstein – somewhat an impossibility. You can only show these things, so it takes several approaches from several directions. But there is actually something like a concise corpus of theory eventually emerging from this. Moreover, this – being transcripts of seminars – addresses an audience that is already familiar with Freud, in order to reframe this. – This is also one of the major issues with Lacan and his reception: it takes some serious investment to get into this, and this also used to have some worth on the academic markets. On the other hand, this (academic) value became indeed inflated and eventually devalued, to the point of those, who never bothered to invest, happily triumphing. Think the Great North-American Video Game Crash. But this really shouldn't be the end to what may be one of the major approaches towards what language actually means to us. The expectation that everything can be addressed directly and without prerequisites, regardless of the complexity, may actually not be met. On the other hand, there will be also never be a single "master", who is always right and without failure, bearing always the most distilled emanation of truth in their very word. – I'm also not arguing that everybody is now to become a scholar of Lacan. Rather, we may have an informed expert discussion, what may gained from this from a current perspective. E.g., if Lacan actually had something to say about an impulse-like directional vector emerging from attention (as a form of selectional focus on a semantic field), is there something to be learned from this, or, to be aware of?)

Thanks much for great pointers and precautions on reading Lacan. Even Rorty finds Lacan, Foucault and friends difficult and he read in French rather than in translations.

I was browsing in the medical school bookstore in Berlin (Humboldt/Charité) looking through the psychiatry section and (not joking) a third of the books were by Lacan. Will try with residual trepidation.

The reading list is already too deep and broad for this mortal. But G. Buzsaki, P. Churchkand, A. Damasio, D. Dennett, M. Donald, J. Hawkins, D. Hofstadter, C. Koch, R. Llinas, M. Minsky, Tommasi, J. Panksepp, Piaget, E. Pöppel … do find good traction for those of us who are neuroscientists interested in levels of compute that lead to language generation by human wetware.

They would be called "unemployable" because they are literal charlatans.

Please end our strange fascination with fashionable nonsense. Freud was wrong. There is no Oedipus complex. Everything lacan proposed was wrong. Deleuze and Guattari's mental health clinic failed spectacularly, and Deleuze ended up killing himself at the end (supposedly due to back pain?)

They literally describe their thought as being "Schizoanalysis". How many more red flags do you need?

Also, the more "modern" takes on this from techno folks, such as from Nick Land (Fanged Noumena), are openly fascist - https://en.wikipedia.org/wiki/Dark_Enlightenment

If you want cultural critique from smart people without it turning into fashionable nonsense, I recommend Mark Fischer, but be warned, he too killed himself.

"There is no Oedipus complex" is, using your own words, "fashionable nonsense". Of course there is a crucial conflict as we learn rules, and this establishes a universal foundation of ethics. And of course this takes place in the realm of our very object relations. Not bothering about what a concept actually means and/or describes is not a free card for criticism. (Neither does generalization over an entire section of science and human thinking help particularly.) – Sorry for the harsh words, but this actually bothers me.

Regarding charlatans, mind that there are already few who have actually studied this. (I'm one of them.)

Regarding Lacan, he provides us with an established theory of "talking machines", and, in a philosophical context, how they relate to our very freedom (or, what freedom may even be). This isn't totally useless in our current situation, and NB, it's actually quite the opposite of fascism.

Guattari was the psychiatrist-analyst type, Deleuze was uninvolved with that. Also: what drove him to suicide was lung problems, to the point he was (old and) locked home connected to a machine to breathe. Some say it was an accident -- that he opened the windows to let some air.

That's just to correct the record. I have no desire to re-litigate Sokal/Bogdanoff and so on. Good day sir cheerio.

In this moment it would be better if people would read some Wittgenstein and Roy Harris, IMHO.

Why not both? Things like philosophy or metapsychology tend to be prismatic, each framework comes with advantages and disadvantages and boundaries of its own.

(A turn towards the dogmatic is something I'm pretty much expecting from the current launch of AI anyway, simply, because the productions systematically favor the semantic center. So it may be worth putting some generality against this, rather than being overly selective.)

Lol well to answer your question literally, I think integrationist linguistics and Wittgenstein's thoughts about language use as a social action are way more relevant to understanding what's happening with LLMs (and people's naive reactions to them) than what was suggested previously as background reading.

Mind that we're are not, by any means, at any state of social interaction with LLMs. (Any such thing would be a mere hallucination on the user's side.) However, these are semantic fields, with whatever consequence comes with this. (So there may have been something said on this already, in what was known as the linguistic turn.)

I would vote for revisiting Hofstadter and perhaps even Piaget. And if we want to bother trying to define and operationalize self-consciousness—-then Guillery’s The Brain as a Tool—is a good foray into how the CNS pulls off this simple trick.

Whether there are demons underneath depends on whether you believe a lossy statistical snapshot across the Jungian subconscious of all human writing contains demons.

It's a bit more plausible when we phrase it that way...

Is there an article that would meet your standard (discusses transformers, uses the word 'attention') that you could give to someone who has no familiarity with computer science?

Jay alammar's Illustrated transformer, although this too is detailed. I think it's still worth taking a look, because really i don't think that people have yet "compressed" what transformers do intuitively. None of the concepts of the networks are particularly hard math - it's basic algebra. But the overall construction is complicated.

https://jalammar.github.io/illustrated-transformer/

The whole thing is magic, really. We assemble relatively mundane things that we understand together in certain specific ways, and somehow get something that can talk etc. A conjurer of old - who would similarly assemble bells and candles - would probably say that we summoned a demon.

Embeddings and their relationship to each other are definitely relevant to transformers. Why do you think that's not the case?

gptX embeddings aren't even words. Even so, the embedding relationship is useful but not the core of what transformers do to find relationships between words in sequences.

remember the word2vec paper? the surprising bit the authors were trying to show was that putting words in some embedding space with an appropriate loss naturally lends enough structure to those words to be able to draw robust, human-interpretable analogies.

I agree with the sentiment that each individual dimension isn't meaningful, and I also feel like it's misleading for the article to frame it that way. But there's a grain of truth: the last step to predicting the output token is to take the dot product between some embedding and all the possible tokens' embeddings (we can interpret the last layer as just a table of token embeddings). Taking dot products in this space are equivalent to comparing the "distance" between the model's proposal and each possible output token. In that space, words like "apple" and "banana" are closer together than they are to "rotisserie chicken," so there is some coarse structure there.

Doing this, we gave the space meaning by the fact that cosine similarity is meaningful proxy for semantic similarity. Individual dimensions aren't meaningful, but distance in this space is.

A stronger article would attempt to replicate the word2vec analogy experiments (imo one of the more fascinating parts of that paper) with GPT's embeddings. I'd love to see if that property holds.

I wouldn't say the interpretability of word2vec embeddings is suprising - it's just a reflection of words being defined by context/usage, and these embeddings being created based on that assumption.

You can't just wave your hand and tell someone that words are broken up into sub-word tokens that are then transformed into a numerical representation to feed to a transformer and expect people to understand what is happening. How is anyone supposed to understand what a transformer does without understanding what the actual inputs are (e.g. word embeddings)? Plus, those embeddings directly related to the self attention scores calculated in the transformer. Understanding what an embedding is is extremely relevant.

Reply on news.ycombinator.com