People with sleep problems are often advised to quit caffeine. Many women who are or who expect to become pregnant also quit over concerns it may harm the fetus.
I'd add mental health/anxiety issues as another reason to avoid it. It can be a fine cognitive enhancer - but caffeine as a crutch will give you a limp.