Just in case it helps even one other person, caffeine was causing me to have severe anxiety. It took me years to realize it could be the caffeine and to try abstaining. It's a subtle and inconsistent link, but there's no doubt in my mind now that caffeine was the cause (or at least an exacerbating factor). I love caffeine and coffee, so I had to prove it to myself repeatedly before finally accepting it.

If you suffer from anxiety, try quitting caffeine. It will suck for several days (in my wife's case a couple of weeks) but if you eliminate or reduce your anxiety, your life will improve more than realize.