I've actually heard this the case for most immigrants, regardless of race. Would you say it's the case in the entire country, or more so the rural areas?
Regarding the first: Yes, I'm sure it exists, including, to some extent, in government representatives. It can express itself in more stark ways in rural areas (If you want to get naturalized and live in a rural village, the village gets to vote on your application; it larger places, the process purely bureaucratic). OTOH Switzerland is not a particularly violent country, so generally the racism does not express itself in physical violence.
The standoffishness is a big cultural thing. When we moved back to Switzerland after 8+ years in California, this was something we had to get used to again. OTOH, as others said, 25% of the population is foreign born, so you can always socialize with those… and, who knows, maybe over time you'll develop a taste yourself for keeping your distance and playing your cards close to your vest ;-)