Also, the biggest party (SVP) often uses very controversial advertisement, which I can easily imagine to be interpreted as xenophobic by foreigners (I'm Swiss myself).
Glad to hear your positive experience in Switzerland - especially now with the passing of the 'Burka ban', seeing Switzerland as xenophobic isn't something that would surprise me too much.
To dismiss a whole people as xenophobic as OP did I think is very much an overgeneralization, but I think a large part of Swiss culture is certainly more xenophobic than even nearby European countries - accepting that is an ugly part of the situation does not mean the Swiss people should be any less proud of what they've built over the years.
Current day exceptions like Donbas, Hong Kong, and Gaza (to name just a few) certainly exist, and there is room for discussion there. There is also an argument that can be held regarding the human and economic cost of getting to a place where the culture fits you.
But to call all attempts to maintain a culture "xenophobic" and therefore associated with a negative... I don't think this is the right way to be.