I wouldn't feel too bad. McWhorter has deliberately picked the fights he's in (see for instance his longstanding feud with Ta-Nehisi Coates) and deliberately associated with IDW-types specifically, I think, to troll leftists. It's too bad, because I think he's got a lot of important stuff to say about figures like Ibram X Kendi.
(Like I said, I think McWhorter is worth paying attention to. Also, he has one of the great podcasts.)