The next person that says textual modalities gets it

Is all work on multimodal systems Let's Map Everything to a Shared Vector Space and See What Happens? Can a cross-modal interface be anything other than a feature map into vector space? Vectors work well for images. That's great. Good job! But if we're now going to start talking about multimodal everything I think we should first iron out what we're doing with language, admit that convenience and efficiency aren't what's most important and stop trying to force language onto something that will never work for it.

Recently I've been considering this in the context of work on document or visual question answering, i.e. the interface between images, the Steph Curry of machine learning research, and written language, ignoring speech/ audio signal processing. If you've interacted with a VQA model then you know it doesn't take much to see they have a long way to go. I think the way to push this forward and work in multimodal AI ("Advertising Intelligence") generally is to reconsider fundamental assumptions about what we're doing. Not endlessly Kaggling for benchmarks against the same datasets with variations on the same techniques for as long as Google and Stanford remain co-conspirators.

If anyone is aware of work here that they consider exemplary I'd love to know about it because all I can find are teams of people annotating nouns with bounding boxes and waxing poetic about textual modalities.