King – man + woman is queen; but why? (2017)

I've seen in readings (and replicated myself on a set of embeddings derived from google books/news) the capital cities:

Berlin - Germany + France = Paris , that sort of thing

The difference is that Word2Vec "learned" these relationships auto-magically from the patterns in the surrounding words in the context in which they appear in written text. Don't forget that this was a revolutionary result at the time, and the actual techniques involved were novel. Word2Vec is the foundation of modern LLMs in many ways.

this completely misses how crazy word2vec is. The model doesn't get told anything about word meanings and relationships and yet the training results in incredibly meaningful representations that capture many properties of these words.

And in reality you can use it in much broader applications than just words. I once threw it onto session data of an online shop with just the visited item_ids one after another for each individual session. (the session is the sentence, the item_id the word) You end up with really powerful embeddings for the items based on how users actually shop. And you can do more by adding other features into the mix. By adding "season_summer/autumn/winter/spring" into the session sentences based on when that session took place you can then project the item_id embeddings onto those season embeddings and get a measure for which items are the most "summer-y" etc.

What happens when you add

  male(philip).

If you're missing that rule then you're getting what you would expect?

Thank you. I knew “woman” is “kvinder” in Danish, but I never made the connection with the English word for “queen”.

Did Japanese get a similar word by coincidence?

There is a nice "all women are queen, literally", https://www.youtube.com/shorts/kOY-vXuAiBA

And a longer text, https://blog.oup.com/2011/10/wife/

Also, gynecology has the same roots.

Another cognate is Classical Greek γυνή, whence gynaecology.

Very much so. It was a little less untrue with older word embedding models* but that kind of semantic linearity never was a thing in practice. Word embedding models try to embed semantically similar words close to each other, but that does not imply linearity at all.

*with transformer models, it is pretty much not even wrong.

I just tried it on qwen3-embedding:8b with a little vibe-coded 100 line script that does the obvious linear math and compares the result to the embeddings of a couple of candidate words using cosine similarity, and it did prefer the expected words. Same 22 candidates for both questions

king - man + woman ≈ queen (0.8510)

   Top similarity
   0.8510   queen
   0.8025   king
   0.7674   princess
   0.7424   woman
   0.7212   queen Elizabeth

Berlin - Germany + France ≈ Paris (0.8786)

    Top similarity
    0.8786   Paris
    0.8309   Berlin
    0.8057   France
    0.7824   London

Sure, 0.85 is not an exact match so things are not exactly linear, and if I dump an entire dictionary in there it might be worse, but the idea very much works

Edit: after running a 100k wordlist through qwen3-embedding:0.6b, the closest matches are:

    king  –  man  +  woman  ≈ queen (0.7873)
    berlin  –  germany  +  france  ≈ paris (0.9038)
    london  –  england  +  france  ≈ paris (0.9137)
    stronger  –  strong  +  weak  ≈ weaker (0.8531)
    stronger  –  strong  +  nation  ≈ country (0.8047)
    walking  –  walk  +  run  ≈ running (0.9098)

So clearly throwing a dictionary at it doesn't break it, the closest matches are still the expected ones. The next closest matches got a lot more interesting too, for example the four closest matches for london – england + france are (in order) paris, strasbourg, bordeaux, marseilles

The meaning is not in the numbers themselves, but in how they each relate to one another.

Also those are not mere numbers here, but vectors. Dimensionality and orthogonality is key to define complex relationships.

This is my problem with people claiming that LLMs "understand". What we usually call "meaning" in intricately related to an encyclopedic knowledge of the world around us. How does this kind of knowledge not get into the same kind of loop you've just described? It is ultimately founded on our direct experience of the world, on sense data. It is ultimately embodied knowledge.

I really think you should actually read the article. None of what you are saying has to do with the content of it, and it will explain how you can do arithmetic with these words.

This is the kind of question that made us doubt that AI would happen one day, cause meaning is quite remote from its expression.

There was always these contextual meanings that differ widely.

Like "carrots are orange" is a fact that's generally okay, but is not true at all, carrots come in a very wide range of colors.

But LLMs completely crushed through these problems. And vector embeddings are a bit part of why it worked.

So yeah, somewhere in those vectors is something that says that when cash is king, "king" has no relationship to monarchy.

It doesn’t work because that’s just wrong. The semantic meaning of “king” is much more than simply “royalty” and “male”. And it will be different for different people based on their experiences and familiarity with English and world history as well.

Then there’s the phonetic angle in addition to the semantic one. Why isn’t cash emperor? Because “cash is king” is alliterative.

Then there’s the orthographic angle: it’s a lot easier to write “king” than “emperor”.

The representation might not need to explicitly encode "meaning", if it does so implicitly by preserving essential properties of how things relate to each other.

For instance, a CAD object has no notion of what an airplane wing or car wheel are, but it can represent those in a way that how a wing relates to a fuselage is captured in numerical simulations. This is because it doesn't mangle the geometry the user wanted to represent ("what it means", in a geometric sense), although it does make it differ in certain ways that are "meaningless" (e.g. spurious small features, errors under tolerances), much like this representation might do with words.

Back to words, how do you define meaning anyways? I believe I was taught what words "mean" by having objects pointed to as a word was repeated: "cat", says the the parent as they point to a cat, "bird", as they point to a bird. Isn't this also equality/correspondence by relative frequency?

    "cash is king". What is queen?

"expenses", obviously. ;)

If Johnny Cash is king, then Queen is queen.

Cash is the Man in Black. Elvis is King.

I asked deepseek and after many options the recommended one was:

Cash flow, "because you need the ongoing stream, not just a pile of cash, to reign successfully".

queen is going to be a funny vector, mostly royalty, slightly chess, a bit gay, a bit rock and roll, a bit bee. Finally:

Queen + One = King

If I had to guess, cash - king + queen = credit (or money or something?). You are just asking the same thing as cash - man + woman, or "What is the feminine version of cash?" because queen - king ~= woman - man.

I say credit, because it is not as physical and direct as cash, so perhaps it is perceptually more feminine?

But I will have to check the next time I work with word2vec.

Can you embed a phrase like that in the same way?

cash - king + queen = cashing, cash - male + female = cashing (in qwen3-embedding:0.6b). Make of that what you will

Perhaps femininity is a more specific concept than masculinity. I call this the "Mrs Pac Man Effect".

It is more common for people to personify objects (say, a rock or a frog or a random internet user) as male than female. In many languages, the plural for a group of things is male even if it only has one male element.

A simple charachter like pac man is male in a universal kiki/bouba sense, but the female equivalent needs a bow (it is a more complex specalization of its male counterpart)

Obviously, biologicially male-ness is a specialization of female-ness, because females more resemble unsexed creatures in their ability to reproduce.

But in the latent space of the human mind, or in language I think male is closer to "default" and female is a specialzation of maleness. Even the words female or woman are modifications of the word male or man.

Perhaps this evolved out of primative social structures. If women occupy a more domestic social role and men a more nomadic one, then you would encounter more men in the outer world. So you would generally associate unknown things in the external world as being masculine, and would associate feminity with specific inter-village inter-family things of your local world

it is about etymology and how tech companys continiously hijack words, definitions, and whole concepts along with the worlds money, power, and authority, the whole move fast and break stuff thing now running up against people who are objecting,pointedly, increasingly with sticks, and perhaps more tellingly, with genuine befuddled disapointment to find the grasping grinding overreach happening in every last tiny purely intelectual nook and crany.