More information on your first sentence in Matthew Walker's Why We Sleep (tl;dr: no caffeine >2pm it's v bad for sleep phases + long half-life).

I'd add mental health/anxiety issues as another reason to avoid it. It can be a fine cognitive enhancer - but caffeine as a crutch will give you a limp.