Yep; it’s obviously not a deal breaker/winner, but we definitely notice. I once joked that we ought to mark up candidates with LaTeX CVs, but knock them slightly down if they stuck with Computer Modern :)
However, the Adobe Utopia font, available by using the fourier package, is so much better IMHO.
The easy way is usually to spot some of the no-no's of layouting; single words at the last line of a paragraph, sentences spanning over page breaks, spacing in block paragraphs, etc.
It's easiest with letters or invoices with some text, where the spacing and word placing is most obvious but you can usually tell in CVs with less text too.
Of course, sometimes Word gets lucky and does a proper layout, but I've found that it's rare that I get to see properly layouted word documents.