Yes, that's fair, I think tolerated is the right word.

I'm tolerated for not speaking Swedish, but I would not be tolerated if I was a criminal, went directly against Swedish values loudly or was a net drain on society in some other way.