I don’t see the association as particularly unnatural: black = dark = scary because we cannot see, and white = light = not scary because we can see. The use of the term ‘black’ as a negative modifier predates its use as a descriptor for race.
It's not like the word "black" is just used with blacklist. We also use it in other places like black market, black hat, and black site. None of these are related to black people, but instead all relate to a shady dark/scary side of things like you mentioned.
When someone pictures a black market they are not thinking something that looks bright and colourful, they are thinking something that is dark and maybe grungy/sketchy looking.
The word policing is something I am really getting sick of.
If it had been black list and green list, I'm not sure anyone would have made a racial connection.
Isn't it the other way around though? Black people skin is not of color black as in actual black. And white people skin is certainly not white.
It's the naming of skin tone that is sloppy and figurative. While the association of white with light/visibility and black with darkness/unknown is natural.
Overall I think the logic of opposing words like "blacklist" just falls apart logically in multiple areas and I think your explanation is a good example of why.