I've found that as I do what I know is my obligation to do, while still reaching for what I feel is my dream, that things seem to almost conspire to move me in routes that actually enhance, rather than detract from, what I really wanted.
Keep the vision of what you want liquid. Go in a general direction. Role with the punches and don't swing at every pitch. Try and use your distinct perspective to give you insights that others do not have.
While the journey might take longer, or may not be as conventional, if you stick at it my experience has been that things work for me when I stay at things regardless of how distant or implausible they may appear at any given point.
I'm only speaking from my own experience, but every seeming downside I've had has, after enough time and experience, actually appeared to be a boost rather than a drag on my long term aspirations.