Came here for this. I hate it too. When I make a site with dark mode, I make sure the dark color loads by default and then changes to light if needed to address this specific issue.