Can't speak for OP but a lot of people's problem with Brexit isn't just the fact we won't be in the EU anymore but the things that come along with that. There's a lot of stuff our government wants to do that the EU has prevented until now. There's also the 'extreme' element in society feeling they can be more open now (racism, hate crimes, etc.).
You can still move to Switzerland as an EU citizen.