Two-Tower Embedding Model | Hacker News Reader

Two-Tower Embedding Model

hopsworks.ai

77 pointsjamesblonde21 comments

I do not want to be too much of a downer, but is there something keeping us from just using the traditional verbiage we've used in the community for years and call it "projecting into a shared latent space" (or something bland but descriptive that a researcher like me could quickly latch onto, like 'separate key and query encoders')?

I understand that proprietary names are necessary to sell ideas, architectures, etc, but projecting things into the same latent space is an....old concept, that, like the old Ecclesiastes verse, come up in new and unique ways/applications. Though there is nothing new under the sun, indeed.

Please forgive this stodgy young person. Frippery and grumpiness are a deep skill of mine, and I apply it to my own DL research as well. I do not want to discourage the author from writing more pieces, explaining concepts is I believe a great trend to have in a community.

Thank you and curious for anyone's thoughts. <3 :) :')))) <3

Two tower isn't a new term afaik. It's often used in the context of recommendation systems where the two modalities are users/query and items.

https://github.com/creyesp/Awesome-recsys

Gotcha, thanks. I guess it could be useful as shorthand for specifying 'joint latent space of multiple modalities', since 'two tower' is shorter and more established (though I guess it had to get established at some point, lol).

Much appreciated on the clarification, something I did not know/had not heard about! Recommendation systems seem to be pretty disjoint from a lot of ML talk/news/etc generally, at least in the sphere of the world that I'm in! <3 :'))))

You are correct that we have two modalities projecting into a shared latent space. However, that doesn't convey anything about how to jointly train the embedding models, does it? That's what the two tower embedding model shows - how to jointly train the models using distance functions like cosine distance.

On an aside, others, as esteemed as Yann Lecun, are talking about learning in a shared latent space between 2 modalities, but use different names - joint embeddings, because it's a different method.

https://arxiv.org/abs/2307.12698

My understanding is the LeCun uses the term joint embedding as an alternate name for siamese networks. It does not necessarily imply multimodality and he has used it mostly in the context of SSL for images.

"Two-tower" or "dual-encoder" is an established terminology, many years old.

I haven't heard the term before either, but it's mentioned at least by Google too so not entirely unheard of in the multimodal training context. Even so I also think this could have been conveyed without using that term. Two towers and even twin towers as seen in some places sounds exceedingly marketingy. But as others have pointed out, this may have been a popular term in recommendation systems before the rise of deep learning multimodality.

https://blog.research.google/2022/06/limoe-learning-multiple...

This is a very cool concept. The example given of Bytedance combining the text embedder ALBERT with the a transformer image embedder, to make an embedder that can do image and text at the same time to get an interaction score is fascinating. I had not heard of being able to combine unrelated embedders before, and I want to know more examples. A quick google search found this in depth blog article of how their using two-tower embeddings at Uber, which was written recently (this past July)

https://www.uber.com/blog/innovative-recommendation-applicat...

Combining embeddings is the backbone of multimodal LLMs, such as InstructBLIP[0] or LLaVA[1]. Those architectures take the output tokens from a (frozen) vision transformer and train a very small projection layer between the output token space of the ViT and the input space of the LLM.

[0] https://arxiv.org/abs/2305.06500 [1] https://llava-vl.github.io/

I wonder when Dalle3 is released if OpenAi will release any technical documents about the integration with gpt4. Their approach might be similar.

OpenAI used this approach for CLIP before blip and llava. CLIP is used to encode the text prompt in stable diffusion. Not sure about Dalle.

The two modalities that will be combined first will be the most profitable modalities :) - Uber combine products with user-query-history-context. Images and text are two other modalities that are getting traction.

I am looking at combining financial transactions and suspicious accounts. You need ground truth data that combines the two modalities - that's the starting point.

So i have played around with this architecture specifically for entity linking. Two BERT encoders: one for the query text and another for the candidates. Initially the two encoders were separate but trained together. When I tried using the same BERT model for both, the accuracy jumped by 4 percentage points. Was pretty suprised by this and i guess it got me thinking that maybe a simple cosine similarity loss function is not enough information for the model to shared latent space. Maybe we also need some weights to be the same between encoders. Granted in my use case above they are the same modality but if we are building a model with image and text encoders it might be helpful to try and tie the weights in the last layers of those two encoders

Check into SBERT, sounds perfect for what you're trying for: same encoder, asymmetric search

I much prefer the n-tower approach https://arxiv.org/pdf/2307.10802.pdf

Soo, if and when this "AI" is applied to suggestions, will it stop to suggest washing machines to me after I just bought one the other day in the same shop? Or the same article I just read this morning? As long as this very simply case is not covered, I consider these algorithms cow manure.

Please don't take this the wrong way, but I'm somewhat knowledgeable about AI and machine learning, and when I heard the phrase "two tower embedding" the first thing I thought of was the tragedy on 9/11. So I'm not sure how good a slogan it is, if that was your aim at all.

It could be a reference to Tolkien's "The Two Towers".

Very true!

Two-tower embedding are fairly common approaches in big tech firms, so you should adjust your perception of how knowledgeable you are about machine learning

Fair enough, haha! I did say somewhat!

Reply on news.ycombinator.com