There are so many things. I am re-writing this answer for 3rd time in hopes of expressing myself in a succinct manner.
I can write pages about it, but I will intentionally cut it short here. Because a lot of the experiences I had are very emotional & vulnerable to me. I wouldn't feel comfortable some random commenter coming and justifying it in all different angles.
There is a HUGE segregation problem in Sweden. Instead of sharing personal anecdotes, I would like to give you an idea how you can observe it. Look around within your Swedish circles, how many expats/immigrants do you have? how many English-speaking ones? Go to a park or a pub, and look around; how many English-speaking tables do you see? Do you observe some ethnical barrier between the tables?
I can barely say I have Swedish friends. I throw parties all the time, I get people from all different nationalities; spanish, italian, french, german. Yet, living in Sweden, I always ask myself "where are the Swedes?". We are not invited to their parties, but they're not even showing up to nomad/expat communities.
Swedish acquaintances (friends) I have; I only see them 2-3 times a year. In the time, I have been here for 7 years, number of times I have been invited to Swedish homes is less than 5.
I indirectly heard; one friend excluded me from a dinner that was to introduce me to their cousin who is also nerdy. Because his girlfriend was uncomfortable with speaking English.
Again, I repeat, these are only some examples. Because once you get anecdotal, people tend to get stuck on specifics and act pedantic.