You're right, they can't. Mastodon accepts that it will have Nazis and other extremists make their own instance to share and proliferate hateful ideas (much as they do on the normal web).
The solution? Block their site.
If you group up all the Nazis in one spot, it makes it very simple to shoot them into space and never see or interact with them again. That's the core moderation philosophy of Mastodon, and IMO it actually works great in practice. Most mainstream instances (1000+ users) will generally have uncharacteristically nice moderation because of this.