The book of mormon makes no claim as to geography. Nobody can point to "Zarahemla" on a map. Joseph Smith was almost certainly thinking of the regions he lived in as the basis for book of mormon geography (north america). You just need to read the story of "Zelph" [1] where he picked up a bone and went on a rant about how it was actually a Lamanite warrior named Zelph that lived there.
The main reason the North/Central America geography were abandoned is because significant archaeologic evidence surfaced describing pretty much everything about the native american population that lived there for 1000s of years (spoilers, no correlation with the BoM story). That's why in roughly the 50->80s the church started pivoting to pointing to south america as the geography of the book of mormon.
The church has spent a lot of time burying stories like "Zelph" because of how bad they make JS look.
> But for a long time people claimed no body even lived there.
Show me one reputable person that has claimed there were no ancient inhabitants of south america.