the first thing that comes to mind is CLIP: https://github.com/openai/CLIP maybe combining it with some sort of knowledge graph is what you're looking for