I think one big reasons why RTS are having trouble is, that loosing in an RTS, as a new player, can be one of the worst experiences in gaming (not counting player toxicity): you just spend several minutes, maybe a lot of them, playing this game, you finally got your T2 fab, and then the enemy nukes you with his experimental artillery and you are just gone. No way to understand what exactly went wrong. To slow? Wrong build order? No clue!
Might you just mismanaged the harvester and you economy suffered early.
Compare that to a shooter, an (a)rpg or a racing game. They might be as hard to win, but at least you have a clear understanding of what happens. He saw you, he shot you. She just had the correct breaking point figured out and you hit the wall.
Ofc it’s not always that clear cut and there is a lot in between these examples.
But in my experience, RTS and maybe strategy in general, can be really hard to understand what went wrong, even on a basic level.
MOBAs are, imho somewhere on between. High skillcap both in regards or game knowledge and strategy, but also in agility, but also somewhat readable.
Sure, you might wonder why your gorillas punch did way less damage than their gorilla, but a look at the KDA or their gear might already give you a hint.
So I guess with the overall growth of the gaming market and games becoming more and more expensive, the small community that actually enjoys RTS isn’t really „worth“ that money.