The hardest thing about giving up caffeine has to be the immediate effects. Your body takes it as a massive shock and you're stuck with what feels like a migraine for 2 days. Not fun.
I know decaf still has some caffeine in it, but it's a compromise for something different – for a month of cold turkey I thought I had cravings for caffeine, but it turned out it was hot drinks I was after. I think it's the one luxury I couldn't be without.