As an American, I think of nothing when people wear hijab or whatever, and trust me, I am an atheist who think religion is irrational and that humanity would be better off without it.
Maybe the French would be better off thinking nothing of it when someone wears religious clothing, just as we are starting to think nothing of it when people are gay or transgender or whatever.