You missed the part where we _voluntarily_ chose to enter into a contract with the EU that does in fact beholden us to EU laws.
We can go back on that contract, but breaking your word is something that people remember for a reason.
And that clause famously includes the breaking of all other contracts.
Freedom of movement for EU citizens. Migrants and asylum seekers don’t have the right to live and work in Switzerland because of our EU agreements.
A migrant or asylum seeker living in Germany has 0 right to move to CH.
We do have some asylum obligations from the Dublin accords and from global human rights laws, but those we can regulate ourselves separately anyway. EU doesn’t care. Countries within EU do it already.