I totally relate to this and it's one of the reasons I try to force myself to go out every day (e.g the office). I don't want to let the anxiety tyrant to grow. Also, being in my mid thirties, I've realised that I've been drinking alcohol since my early twenties (nothing crazy, just a few pints of beer every now and then) not because I like the feeling of being drunk, but because I hate the feeling of being anxious and alcohol, at least for me, acts as a solid anti-anxiety drug. Anxiety sucks because it affects critical aspects of my life (social life, sex, perceptions about life itself etc)
It's a complex and annoying problem, because if I go to the GP, I might end up taking pills and I really don't want that. The alternative is to go to a phycologist but, as with restaurants, it's better to get a recommended one and I haven't got any where I live.
Thanks for sharing your thoughts.