Do you feel the same about people who believe in a religion other than yours (or any religion if you're not a believer)?
I'd always thought the primary difference between vegetarians and vegans was ethical, rather than dietary. That part of the ethics of veganism is opposition to the mechanised slaughter of individuals that should have the same, or similar, rights to people.
Maybe this is more of a fringe belief as vegan options become mainstream?