- There needs to be a lot of you on a lot of horses. This is pretty hard to achieve especially if most people are subsistence farmers. An infantry formation is much easier to conjour from a farming community. To get cavalry you need feudalism, as Anglo Saxon England discovered, much too its cost.
- You need the right horses. Little horses are no good. War horses are the thing. War horses are no good for agriculture and as soon as cavalry stopped being a shock battle winner and switched to being a battle ender instead war horses went extinct. Because breeding them and training them was staggeringly expensive.
- You need the right terrain. Forests are not a good place for cavalry. Roads are needed to shift cavalry (and fodder) around.
- Logistics; a knight needs a riding horse a squire on a riding horse and at least one war horse, and a cart, with fodder, and two donkeys to pull it, and a cartman. You need to have farriers with the formation. Suddenly sustaining your army in the field is much more difficult.
If neither side can do these things then stirrups will not be important. If one side can do it and the other can't then they might be important unless...
1. Castles - if the opposition go and sit in a castle then good luck with the cavalry thing.
2. Long bows - on occasion cavalry forces faced long bows and things went very badly. Not always, on some occasions the archers weren't well placed and got jumped and made very dead, but an archer is much easier to replace than a knight, warhorse, riding horse, squire, riding horse, donkey, donkey, cart, cartman unit.