I wouldn't really say that passports are a very recent thing in the history of travel, since passports predate mass travel. The expansion of rail in 19th century Europe made enforcing passport requirements much more difficult. But even before passports, people would travel with letters of recommendation or other such papers to indicate who they were. While the document called a 'passport' has only been with us for a couple of centuries, the concept has been around for ages.

And while you can theoretically travel without a passport, don't expect an expedient process on arrival. Lack of a passport is not a trivial thing.

Because border control per se is a rather modern thing.