I think the difficulty most people have is that Stallman is, or at least comes across as, an uncompromising idealist. Whilst there's nothing wrong with that (we need some people to stick to their guns), most of us are pragmatists who have to compromise regularly in order to get things done, so someone who appears to refuse to change his position and shoots down anyone who disagrees with him can be very annoying.

Compare Stallman with Torvalds, who seems to be much more pragmatic and willing to listen to people who may have different views, and seems to be much more popular as a result.