I agree with your whole comment, but the media query approach is also tricky.
You typically still want to offer a toggle while respecting prefers-color-scheme, and you may want to give priority to whatever choice the user has made with that toggle if they've used it on your site before. This still requires JS and localstorage.
So what to do? My preferred choice is having an inline style element of:
body {visibility: hidden; opacity: 0;}
This will maintain the previous page until DOMContentLoaded, at which point you can query prefers-color-element and localstorage, set the body class correctly, and let the page paint.