I dispute this. And now this is going to be cited, further manufacturing the idea that this is an empirical truth.
I think people probably find situations where one variant of dark mode is just less bad then one variant of light mode, and that perpetuates the simpler idea that dark mode is always better than light mode, where light mode is often unfairly represented by black text on white.
I do admit that it can be more complex to tune your setup for non dark modes. And sometimes it's officially impossible, like when browsers don't support tweaking your own colors well. But I think the popularity of dark mode prevents this kind of tuning from being more popular and supported.