High immigration(besides nearby catastrophes) usually means something good happended to the area. So people can have the same overall happiness but for different reasons.
For example: Job opportunities are better now, Property prices are rising(Great my home is worth more), but it feels less at home then before.
I live in Munich. Aside from exorbitant rents caused by politicians ignoring housing issues for decades, we don't have a significant problem w/ immigrants and refugees. We do have a couple Nazi groups (two different Pegida groups, plus the "old" NPD nazi party plus the AfD neo-fash) with maybe two dozen people on the streets, but no one except Antifa gives a flying f..k about them.
Also i personally dont know anybody who got negatively affected (besides closed school Gyms) by the refugee crisis, but many worry anyways. Probaly Social Media makes it feel worse then it is,similar to people feeling less safe now despite way lower crime rates.