If you want to hop in a boat and motor away, no one will stop you. But if you leave this country, you probably want to go into another country (the world is about out of wilderness), and what are you going to do when you get there, without a passport?