LSD does cause significant psychoactive effects with one thousandth of that amount, which suggests that for a typical isolated dose, LSD is safer than caffeine and amphetamines. Not to mention that LSD is not as addictive.
But I don't think that invalidates my underlying point regarding caffeine, even if the lethal dose is not a perfect measure.