For Org Mode, my main concern is that I want to quickly identify the structure of my documents and differentiate all the relevant parts visually. I like to keep font sizes and spacings small to fit as much text (= context) on my screen as possible. I also like using monospace fonts because it feels more like working on a draft (e.g. with a typewriter) to me and I can work more spatially with text (e.g. making little diagrams or figurative text arrangements) which would not be possible with proportional fonts.
When everything looks too good/refined, I get distracted by wanting to shape my text to make it fit in that perfect environment. Then there is no space for swift notes, unfinished drafts, rough edges and this becomes a problem, since I am already a compulsive perfectionist and like to waste a lot of time with irrelevant details. Of course, this is just my personal preference, so I am not saying that this is how it should be. Maybe I am just trying to understand why I like it better this way.