A federated system of isolated instances doesn't solve it, because that's not a federated system. The point of federation is that the instances aren't isolated, and they're highly interchangeable. No major instance has the incentive to become a new Reddit, because it's so easy to switch instances at every level that they just don't have the moat to make that happen. It's exactly the same thing as an email provider going to shit, but even less problematic.
If a Voat instance pops up, sure, dominant instances won't federate with it. That's perfectly fine. It doesn't mean that the inter-federated dominant instances can get away with pulling a Reddit.
Federation is an opt-in feature in theory, but in practice, what's the value proposition for an instance to turn off federation today?
It's really quite simple. Without federation, there is an incentive to pull a Reddit. With federation, there is no longer an incentive to do so, because you don't have a moat. What do you think would happen if Reddit only had 1/2 of the subreddits anyone used and if you could keep access to all the same communities on a competitor?