The people one state over are part of the same compact (the Constitution) with you. A compact which, by the way, bars you from discriminating against them in commerce, which actually causes a lot of problems in practice. E.g. states that want to enact social welfare programs for their own people have a difficult time doing so at the state level, because of free rider effects (it is unconstitutional to require a citizen of another state to live in your state for say a year before applying for welfare benefits). Trying to get around the free rider effect creates increased demand for solutions at the federal level, which undermines federalism and state sovereignty. At the level of sovereign nations, similar pushes for extending rights across national boundaries creates pressure for trans-national governments. It's no accident that the tearing down of economic barriers between European states went hand-in-hand with the creation of trans-EU legislative and executive bodies.
There is a lot of value to trans-national barriers that isn't immediately obvious, especially in a democracy.