typography. it's not the only thing a good designer knows, but it's a basic skill that the good ones respect.
there are 2 schools of design that a savage like me can comprehend: systemic and emotional. i'm sure that many can argue for the many gradations, but these are the important distinctions/extremes to recognize. that said, it's not either or - successful designers evoke (that's an important word- what does your potential designer's portfolio evoke) their ideas in a consistent/lasting manner. this requires both schools of thought - powerful ideas that can be repeated/propagated.
emotional design is provocation. vimeo is better at this than google - compare the persistent vimeo login page with the occasional google holiday logo. that said, there's a much wider range of emotions that neither of these places touch. good emotional graphic designers will understand how to communicate your product evoking a positive emotional response.
systemic graphic design is the ability to maintain an identity, consistently, and create familiarity. most of us feel that if we were hired as a brand manager for google, we'd have a pretty good sense of whether or not a page put in front of us meets their identity's requirements. that's a test to remember. more than any one event in your brand's lifecycle, systemic graphic design will be there to give you a (relatively) easy solution in any situation, once your voice has been defined. best example in a typeface: helvetica. these designers demonstrate a comprehensive understanding of systems design. look at Paul Rand's work & Josef Müller-Brockmann to see evidence of a system that's designed to be familiar, even when it's radical.
ideally, you need both. an easy metric that would hold up against the best ones i know (and no, i won't release their names, they're busy ;) - they love to talk about their art and they will not hesitate to tackle a new challenge, especially if it seems hard. even at the expense of dayjobs & prior commitments. then, give them space. when ideas seem ethereal, be very clear on your business requirements, but let them solve the problem/bring it back to earth.
hope this helps. graphic design is half of how i discovered (social) hacking and i can't stress its importance enough - it's a useful field to learn about, not just depend on.