You can be certain that there are plenty of Swedes that see it differently, they are just less vocal
This is specifically the case with the Sweden Democrats. A big motivator for them is a) bashing greens, and b) being a better alternative for the working class than the Social Democrats. Siding with a foreign billionaire over "salt-of-the-earth" workers is not a winning move.
Vocal or not, no one will not be able to affect this semi-voluntary system unless they control bascially the entire economy. This is unlikely to ever be the case, until maybe some supranational EU agreement changes the rules entirely.