In my experience, quitting caffeine is much harder than quitting weed. I've managed to go from daily use to no use for years with marijuana, and did not notice any withdrawal symptoms. Caffeine on the other hand I have tried numerous times to quit, and failed every time. The longest I've managed to go is 3 months, and the withdrawals are terrible.