Genetics play an important factor. CYP1A2, one of the enzymes that metabolizes caffeine, varies by as much as 40x between individuals.
In those with a high level of the enzyme, caffeine is metabolized quickly, so they can drink coffee in the afternoon and still fall asleep. They are less likely to get anxiety from caffeine, and do not suffer from a higher risk of heart attacks with coffee consumption.
In contrast, a small amount of CYPA12 prevents caffeine from being metabolized and prolongs its effects.