Web coding is not rocket science, but good web design is like good engineering. Good tools will help, but knowing what to do will help a lot more, and that's what you pay the web designer for. Saying "with all the tools and stuff available to designers today, there is no reason a site should cost more than a few thousand" is like saying "with all the IDEs and stuff available to coders today, there is no reason a program should cost more than a few thousand." Programmers know that that's BS; web designers feel the same way.

Unfortunately, it's a lot like hiring a programmer in other ways -- the best designers will easily be ten times as good as the worst ones, and there's no necessary correlation between skill and rate.

So returning to the original question: look at the web designer's portfolio. Ask questions about web standards and (X)HTML compliance, usability, A/B testing, cross-platform CSS issues, readability, copywriting, audience. A good designer will understand these things and be able to explain them to you in terms you'll understand. Make sure you turn on your BS detector too, though, as a bad designer will try to baffle you with buzzwords.

And one of my favorite interview questions, when looking at a designer's portfolio, is "Why doesn't Site X validate?" The point is not to get the right answer, because there isn't one, but to figure out how the designer approaches these tradeoffs. I've also never interviewed a web designer who provided a portfolio of only valid HTML and CSS.