It's some form of uncanny valley type of thing where every detail or facet or property I might point out, there are other people exhibiting the same nominal qualities without bothering me. But something here is just slightly off in some way, but so subtly you can't say what it is.
I honestly can't say he's doing anything actually wrong. I don't think even the most dumbed down ones are really guilty of being wrong. Not enough that I'd complain about it in a casual yt video.
I can see some people maybe not liking his basic style of exposition. That slightly affected dreamy pondering thing.
One thing I can observe is that another guy that just turns me right off even worse is the Smarter Every Day, and these two appear in each others videos at least a few times. Or maybe it was just one time and that one time just bugged me that much :)
I don't know what it is but I frequently see a video from either of these guys where the topic they are presenting looks interesting, but then I do not like their video on that topic.
I think it's down to the presentation just seems a bit too affected. And it's not even as bad as seeming fake or insincere. They both actually strike me as perfectly sincere.