"From various incidents that have come out it seems as if it's the Wild West where whatever the idiot harassing you decides, with little recourse if you disagree"
you're right to a large extent.
https://www.aclu.org/other/constitution-100-mile-border-zone