In South Africa during apartheid, Bantu was used as a synonym for Black. The "self governing" regions were known as Bantustans and so on.

Like with most words, Bantu has a neutral actual meaning, but it's been corrupted by use and would not typically be used by a polite (white) person today - at least in Southern Africa.

Beyond that, "Bantu" is a colonial term, coined by German colonist Wilhelm Bleek. Bleek has been described as "the first serious thinker about and systematic theorist of ‘race’ in colonial South Africa." He studied African languages but called them "primitive", comparing them to the communication of non-human primates.

Not surprisingly given this history, the word "Bantu" was notoriously used by the apartheid government of South Africa as a catch-all term for indigenous people, regardless of their actual ethnicity.

But somehow, linguists haven't gotten any of these memos and continue to use the term to describe a family of languages. That may be because Western linguistics has deeply racist roots:

> Western linguistics particularly its study of Eurasian languages, arose against a background of Eurocentrism, colonial racism, nationalism and related theories, later espoused by Nazism and other White Supremacy movements. The following article argues that significant traces of this racism remain in contemporary Indology and South Asian lingustics—casting a long shadow over how the world is viewed.

-- https://www.jstor.org/stable/48505004