I take two weeks off from drinking coffee each year (not because of the coffee, it just works out that way).
I had horrible withdrawal the first year, and switched from 3ish cups of American style coffee to 3ish shots of espresso with water or milk. Same amount of liquid, but considerably less caffeine. I haven't had withdrawal problems since.
After only a few months I noticed my vision was getting blurry, lacking focus, and there was some visual swirling as if I was on LSD. Fast-forward another 9 months and colors were muted, all emotions were numbed, I was insanely irritable, and I felt completely distant from the world. I felt so numb to life that I was very close to seeking therapy due to increased thoughts of suicide. On weekends I never had caffeine, but one day I decided to try some tea. That little amount of caffeine was enough to kick me out of the withdrawals and and make me realize it was the caffeine that was causing it.
It's been two months since I last had caffeine and I'm still going through withdrawals. For a while I had to take ibuprofen because the headaches were so severe, but now it's just a slight head pressure. Colors are popping, sunlight is gorgeous, I can focus on words in a sentence without my vision darting around. It's better, but I know there's still another month or two to fully reset. I'm getting back to feeling like I did in grade school though which is amazing to me. I've always drank large amounts of soda since middle school/high school, so I've likely always been impacted by caffeine like this at a smaller scale.
I had actually tried stopping early on, but was sticking with decaf and only for two weeks which wasn't long enough to notice anything.
For some added info, I have the A;C form of http://www.snpedia.com/index.php/Rs762551 which supposedly makes me metabolize caffeine more slowly.