i draw the line at "is the substance itself acting as a neurotransmitter in your brain". definitely sex exercise diet &c are indirectly drugs, in that they inspire your body to adjust its own mix of chemicals in the brain, but it's a whole 'nother thing to directly inject substances into the mix yourself.
i think where people tend to get into trouble with drugs is that since it is an external effect changing the brain's chemical balances contrary to its self-regularted levels, the brain tries to adapt, leading to tolerance – the user tries to counter-adapt, upping the dosages, "chasing the high", & often ending up in a no-fun place of dependence, addiction, side-effects...
(obviously more true w/ some drugs & some people than others...)