It's fundamentally a bit of a gamble. The folks that keep doing it enjoy the gamble. There are some hills I've crested while bikepacking and some towns I've been through that have been absolutely gorgeous and those certainly help, but everyone I know who enjoys this kind of travel enjoys getting outside and getting a bit rough and dirty. I encourage everyone to travel like this as long as they feel safe (and it's important to remember that different genders, sexual minorities, and ethnicities will feel differing levels of discomfort in different locations so safety should always be top priority.) But if you don't like the gamble, then head to a resort and take a tour bus to the Taj Mahal!
You'll also meet a network of longer-term travelers who exchange info about where to go (and not go). You don't have to go somewhere that literally no one has been to see something a bit unique, magical, and mostly devoid of casual tourists.
I suppose it's not realistic to do this for a lot of destinations, but where it works it works well.
I now know well a tiny village I go back to regularly where I'm part of the small summer swell of visitors, to go fishing and sailing without a crowd of tourists, because getting there is a bit complicated without assistance.