I think it has to do with Wittgenstein's constructivism. Since humans can never actually construct an infinite set, one cannot say that such a thing exists as a constructivist. As such, infinity would be a fuzzy without-size concept that Witty was suspicious of. But if one does think mathematical objects are real, then it's not a problem.

I'm guessing constructivists have found better ways to approach infinity than Witty, without conceding that infinity is real.