Nice work. The authors train a model with two contrastive objectives: (1) predict which video clip corresponds to a given audio file, and (2) predict which audio clip corresponds to a given video clip. Together, these two training objectives induce the model to learn to associate different sounds from a given audio clip, like words forming a spoken sentence, with different objects shown in a video clip, like the pixels depicting the subject of the spoken sentence -- and vice versa.

If I understand the paper correctly, the model was trained on tens of thousands of video and audio clip pairs, randomly sampled 64 million times (80 samples/batch x 0.8 million training steps). For comparison, over the course of a baby's first two years of life, the baby will get trillions of sound samples (measured at 44.1KHz) and billions of frames of visual data (measured at 30fps). It's not too hard to imagine that a large AI model in the near future, trained on comparably vast quantities of audio and video data, can learn to recognize objects and words, like a baby.

The most exciting takeaway, for me, is that a bigger model inside a robot body should be able to integrate learning in this manner across all five human data modalities -- vision, hearing, smell, touch, taste -- as well as any other data modalities for which the robot has sensors -- radar, lidar, GPS position, etc. We sure live in exciting times!

Link to paper, code, demo, and data:

https://mhamilton.net/denseav