New research shows that by extending instruction tuning to handle visual tokens, LLMs can simultaneously learn image understanding and generation with minimal changes. The most intriguing finding is that visual generation capabilities emerge naturally as the model gets better at understanding - requiring only ~200K samples compared to millions typically needed.
It suggests current LLM architectures might already contain the building blocks needed for unified multimodal AI.