It's not, but insisting on using it even though there are perfectly fine alternatives and you know that there's a sizeable chunk of people that considers it racist is just being an asshole.
Remember the whole ordeal with the "ok" hand sign? That's what happens when you let the minority control you, and that wasn't even a real case of racism.
You see, the thing about culture is that it's rapidly evolving and not necessarily rational (which is perfectly fine). If the word "mainline" would become associated with nazis commonly enough, I'd definitely want to avoid its usage despite of there being nothing wrong it, just like I'm not going to use swastikas just because it's an ancient symbol of well-being. So no, "blacklist" isn't racist, but if we can reduce the ugly associations that our language may be implying, then insisting on not doing that surely doesn't present you in a good light.
What's this "sizeable chunk"? Where do these people congregate?
That is dangerous overstatement.
That is a hyperbolic statement.
"Blacklist is racist" is indeed a dangerous overstatement.
This is totally untrue and everybody knows that.