Please look up studies by Mathew Walker. They did numerous studies with people claiming coffee did not affect their sleep. What they found was while those people would fall asleep, and sleep throughout the night, the quality of the sleep (i.e. minutes in deep sleep) was substantially worse.

But I agree that abuse is what is causing majority of the problems (and it was in my personal case)