This may be hard to answer, but how far into the conversation do you usually know whether he/she's a match for the job or not?
You can speed things up by knowing what to look for ahead of time-- (skills, experience, motivation, cultural fit) and having a formalized candidate scorecard.
Use a stepped process where you either move an individual forward or eliminate them. First conversation via phone/Skype. 2nd conversation in-person at the office, meet the team. 3rd conversation-- meet for lunch/dinner. By this point you should know. Make a decision, don't drag it out.