They are both drugs, but the side effects of caffeine are a lot less bad per usage compared to weed per usage. Perhaps I should phrase that a bit better, but I hope you see what I'm getting at.
Caffeine can have pretty "bad" effects - issues with sleep, headaches, panic attacks, and all sorts of cardiovascular issues (increased heart rate, sweating, etc).