How do you politely turn down someone who wants to "grab coffee sometime"?

I always thought the suggestion of "grabbing coffee" was just a polite way of brushing me off - ergo no response is really needed. Perhaps I'm just dense, tho'.