It's as hard for the person with the dark colour as it is for everyone else. Most people are reasonable if you ask them to change their colour, too.

Otherwise you can kick them between rounds.