I do think it cuts both ways; veganism is extremely strange. The strangeness does not prove that it is good or bad -- but absolute veganism (as perhaps distinguishable from other historically popular, frequently religious diet practices) is wildly out of line with nearly all humans in all of human history -- and could reasonably appear to be overly simplistic.
So, I think the "vegans that are complained about" are nearly always the folks who don't much take this into account.