Very cool! I was briefly toying with something similar just last week.

Main challenge I see is character consistency. I really like the way you set up the prompts, but even so:

Outfit - a simple black sleeveless gi with white pants, a black belt tied around his waist

In two consecutive panels, the output swaps the colors (first panel gets it right, second panel has white gi, black pants).

Curious how you'll tackle this challenge!