The things you've said certainly make sense but I feel like much of that also applies to political ideology. It certainly seems to me like much of that ideology is at least partly subject to location and culture. In particular I feel that a large part of our individual ideologies are based on our past experiences and the way those experiences shape our perspective of the world at large.

Edit: If we're going to use the logic that religious views should be protected due to their inherent "stickiness" relating to family and culture, and other aspects out of one's immediate control, I'm not convinced that other ideological views are any less "sticky" and thus not equally worthy of protection.