It’s like serious literature has to eschew anything that would make it a good story, in case it got in the way of being good writing.
On the other hand, one could argue that if your works require elaborate setpieces and a fantastical setting then you're drawing attention away from an inability to write captivating stories. A plot summary of a Jonathan Franzen novel for example sounds incredibly boring but his writing is nearly impossible to tear yourself away from and will have your mind going for days or even weeks after you finish. It's perfectly fine to enjoy sci-fi/fantasy etc but there's no need to develop a victim complex about its position in the literary world.
I found Franzen terrible FWIW, but there is certainly excellent literary writing out there. Prosecraft is a skill, just as plotting and settings are - neither is superior to the other. And there is a real cultural disrespect for the things that SF does well. I wouldn't mind that if good SF was still allowed to continue in its own place, but lately even that seems under threat (e.g. the Hugo awards are lately dominated by literary prose at the expense of the classic SF virtues).