This is state by state. In Iowa (and only Iowa to my knowledge) the state does have land registration which is worth something in court. In Iowa when property changes hand the legal record changes in a way that is easy to verify and holds up in court. Attempts at fraud are rejected by the state because they can and do verify that the seller has legal right to the property. If someone else has paperwork saying they own your land not court fight is needed because the state holds the final word on paperwork. (even then I suppose you can get identify theft fraud to transfer ownership, but this is harder to pull off than writing up a fictional title)

Even in states without the above, there are local records of ownership that you can bring to court that can be brought to court to prove who is right. Someone is going to be paying the land taxes and that is strong evidence of ownership in court. Banks require title insurance because the title insurance company will check the local records to verify that who you are buying from has rights to sell. However this will be an expensive court case as you say. Fortunately such things are rare these days.

Iowa has their system because about the time they became a state there was a big problem with fraud where someone would sell fake titles to land to multiple people. So Iowa put in place a system to stop it. Other states started when such fraud was rare and didn't need it, or started after title insurance was invented to take care of the fraud problems.

(Note, others have talked about squatters rights - I didn't touch on that, but it does exist in some cases)