Unless your provider really sucks, it's always important to evaluate carefully and ideally stick with them unless it is really that bad.
Yes, that's exactly my point: At some point you are completely locked in with a single supplier that holds your entire income stream and perhaps even more than that.
Ideally (easier said that done, I know) you want to have at least 2 suppliers for any key piece of infrastructure as early as possible and to avoid letting a supplier 'expand' the number of tasks they do for you too much.
The latter seems to be Stripe's strategy: They start with payments then expand step by step in everything related et even in things like company incorporation.