Meta AI: Fist high-performance self-supervised algorithm for multiple modalities

I imagine a multimodal platform like Facebook (with people using photos and text together in some way) would find a single model valuable.

It has already been public on their repository, fairseq.

The benefits are similar to when you consolidate tasks with a lot of similarities. Advancements in one area can be immediately applied to another area. You can consolidate your effort into a single point. Training a network is increasingly cost prohibitive, into the millions of dollars, and this would allow you to consolidate these expenses. EDIT: Looks like they still need to train for each modality so no benefit from that here.

I'd imagine on the flip side the potential problems would be when gains in one modality are offset by a decrease in performance in another or you are prevented from trying new things because of the choices made to support a unified architecture.

Validation is different from supervision. Supervision happens during training. Typically when we train speech model, we prepare a speech segment and its transcript, this transcript is generated manually by human. For language models such as BERT or GPT-3, you don't really need human generated labels, what it does is to mask out some text, and the masked out text is the label to be predicted (self-supervise), the label is already in the data. What this model does is to universally train on images, text and speech this way, and when you combine these data, they will be complementary to each other. Think what you can do with a language model like GPT-3 now, you would be able to ask the model to do something with text, speech and images. The only thing I would worry about this model is that I would imagine Facebook trained on lots of data, and the model may be used to extract sensitive data maybe.

The multi-modal part is a big deal because we could have a model be able to understand all inputs human needs.

I work on ML recommendation algos for a big Facebook-sized company, so perhaps I can give some insight on how this model could be used. This is an example and might not resemble anything they're doing internally. This is more ELI16:

Facebook marketplace:

- Seller posts a listing with image + description

- FB uses data2vec to transform the image + description into one single vector (e.g. you could average the two vectors)

- Buyer searches for a product using text search - the text is also encoded into a vector with data2vec

- To serve product search results, you find the closest match to the text query vector from your pool of product vectors

This pattern described above is generally called vector search and is very common for recommendation algorithms and much more. There's a shift towards algorithms like data2vec that can combine different types of data into one vector. The aim is that an image of a dog and the word "dog" would map to the same vector, meaning that the vector represents the concept of a dog, regardless of the input data type.

The advantage of these "multi-modal" algorithms (i.e. they can take multiple data types as input) is that you can (in theory) use them across all of your ML algorithm needs. If you're Facebook, you have 100s of teams and services that have this need. A few examples:

- Instagram ads prioritization

- Instagram search

- Harmful content moderation

- Facebook content search

- Facebook marketplace search

Each of these is likely a separate team, very likely using a separate embedding algorithm. As approaches like data2vec improve, there will be some consolidation.

N.B. - I've made a lot of assumptions based on what I've seen at my current employer. If anyone from Meta/Facebook is reading this, please chime in!

I think the big achievement is how it surpassed in performance previous models for each individual modality.

That's not quite how I understood it

> Do the cutout in feature space, not the original input space.

I think they do the cutout in the original input space, based on examples they show, e.g. they mask parts of text and grey out parts of an image.

> and make the network predict the missing part

I think predict the latent representation (as you say in (2)), but not of the missing part, but of the whole corrupted sample, and require it to be close to latent representation of the original input done by the teacher.

I believe that your interpretation is not correct. Based on my brief reading of the paper, the model contains 1) some known architecture for embedding the modality, and 2) the feature reconstruction transformer network, the two being trained at the same time.

If I am not mistaken, the masking occurs in the input modality, not the feature-space, even though it is the feature-space that is used for the reconstruction task.

Regarding how the feature space is kept uncollapsed, it seems like a hyperparameter-tweaking (ie unsolved?) problem; quoting the paper:

" Representation collapse.

A common issue with algorithms which create and predict their own targets is representation collapse. This occurs when the model produces very similar representations for all masked segments making the problem trivial to solve. Different strategies have been proposed to address this issue, e.g., contrastive models such as wav2vec 2.0 (Baevski et al., 2020b) use the same target representation both as a positive and a negative example, preventing collapse. Algorithms such as BYOL (Grill et al., 2020) do not optimize the teacher parameters to minimize the loss. VicReg (Bardes et al., 2021) adds an explicit loss encouraging variance among different representations.

In our experiments we found that collapse is most likely to happen in the following scenarios:

First, the learning rate is too large or the learning rate warmup is too short which can often be solved by tuning the respective hyper-parameters.

Second, the EMA decay rate is too low which leads to student model collapse which is propagated to the teacher due to parameter tracking. This can be addressed by carefully tuning τ0, τe and τn.

Third, we found collapse to be more likely for modalities where adjacent targets are very correlated and where longer spans need to be masked, such as for speech. We address this by either explicitly penalizing the lack of variance (Bardes et al., 2021), or by promoting variance through normalizing target representations over the current sequence or batch (Grill et al., 2020). The former worked well for small models but is less reliable for larger models and it also requires tuning additional hyper-parameters. In contrast, we found applying instance or batch normalization before or after averaging targets to work well while being simpler. For models where targets are less correlated such as for vision and NLP, momentum tracking is sufficient to prevent representation collapse. "

Before doing that, it also misspells "first".

At first I thought it was about an algorithm to generate text with mistakes on purpose - and that it did a nice job.

Can you clarify a little bit on what you mean by your question?

One area of research is extracting an effective type theory from data that is viewed as a semantic model, eg sensor data of a phenomenon would lead to a type theory describing it.

You’re essentially taking a TDA persistent homology/covering and reinterpreting that through the lens of homotopy type theory to “decompile” your data. There’s some early results, eg connecting convolutions and type division.

But that’s overall at really early stages of research.

Crazy, but who thinks AI is not moving forward? I don't think anyone on HN.

> Similar to our work, both BYOL (Grill et al., 2020) and DINO (Caron et al., 2021) regress neural network representations of a momentum encoder, but our work differs in that it uses a masked prediction task and we regress multiple neural network layer representations instead of just the top layer which we find to be more effective. Moreover, we demonstrate that our approach works for multiple modalities.

From the related works section of the paper.

"Our work does not perform multimodal training but aims to unifiy (sic) the learning objective for self-supervised learning in different modalities."

No, they don't need a pre-trained model, they start from scratch. Also it's only the approach which is common between modalities, they train them separately.