I think the issue is more or less political, and it's twofold.
The first is that Donald Trump explicitly ran a primary campaign that at one time called for banning all Muslim travel to the United States. It is difficult for me to see how that promise is not rooted in some combination of bigotry and ignorance. For those who remember such promises Trump made, Trump will not automatically signal "compassion" and "tolerance", and I'm sure it's easy to morph an awfully crude executive order targeting countries already on a watch list into a policy that advocates "racism" and "bigotry".
The second is that the executive order was so poorly constructed, so hastily issued, and so overly sweeping, that it snagged people that shouldn't have been snagged (your green card holders and already-accepted refugees and whatnot) and threw immigration into chaos. Again, this only adds to the overall impression of "racism" for those who already have it.
From my perspective, my impression of this Trump executive order is more one of shoot-from-the-hip incompetence (that could've been avoided merely with more consultation with policy experts) and a fairly deaf political ear.