That would be like blaming DALL-E weirdness on GPT-4.
Unfortunately, Google marketing decided to slap the "Gemini" brand on both the end-user interface used to interact with the model AND the actual model itself, hence people constantly calling out Gemini-the-model for weird decisions made as part of Gemini-the-user-interface.
Actually when you trigger DALL-E through GPT-4 (i.e. with the LLM generating the prompt to give the diffusion model then returning the resulting image to the user) the LLM's system instructions [1] say "7. Diversify depictions of ALL images with people to always include always DESCENT and GENDER for EACH person using direct terms." and a bunch of stuff along those lines.
In OpenAI's system this doesn't always trigger; if the user asks for an image of trash being collected, the user hasn't explicitly asked for any people to be depicted, so the LLM doesn't find anything in the prompt that needs diversity added. The trash-being-collected prompt gets passed to DALL-E unmodified, and the resulting image has all male workers.
[1] https://raw.githubusercontent.com/spdustin/ChatGPT-AutoExper...
Again, that's not a GPT-4 thing: that's a ChatGPT interface running GPT-4 with DALL-E as a tool thing.
The way I read the Gemini technical report, it seemed like, unlike GPT-4 vs DALL-E, Gemini was pretrained with multimodal outputs. Is that not the case?
The paper here https://arxiv.org/pdf/2403.05530.pdf has a model card for Gemini 1.5 Pro that says:
Output(s): Generated text in response to the input
(e.g., an answer to the question, a summary of
multiple documents, comparing documents/videos).That feels like it runs counter to this statement from the Gemini 1.0 technical report[0]:
> Gemini models are trained to accommodate textual input interleaved with a wide variety of audio and visual inputs, such as natural images, charts, screenshots, PDFs, and videos, and they can produce text and image outputs