I've learned a lot about the mechanism of establishing friendships, though. As niftich points out above, I had the tendency to consider friendly encounters as one-offs; more specifically, "Oh, that happened. How pleasant." And then go through the mental acrobatics of wondering if the person would like to continue being friendly or [s]he was simply being polite. So the idea of "broken fantasy world" works in reverse here: I have a broken core belief where I don't understand the value someone gets out of hanging out with me.
Just a week ago, friends came over to celebrate my birthday and watch the WFTDA champs. Another friend who is not part of that circle remarked on something that hadn't really clicked for me -- I must be a good friend (and loved) because my house was full of people. So the feeling of loneliness, somehow baked into a bunch of erroneous core beliefs, also masks reality and hinders the formation of relationships.
[1] http://www.ibblaw.co.uk/insights/blog/boarding-school-syndro...
I don't think this is a "forever" state as you put it, but I think this is why some people suffer for so long (some researchers call this chronic loneliness [0]), and turn to fantasizing friendships as a way to cope.
I want to believe there's a way out, I'm not certain, but I think therapy and mindfulness could help, and also being around secure/empathetic people. Loneliness is kicked around as the bastard child of psychology and most practitioners will disregard a lonely person as simply depressed.
But taken to an extreme, according to attachment theory, there is a class of people who naturally gravitate to social isolation yet, paradoxically, desire closeness -- they're called avoidant personality types [1]. There is early research (Bowlby, Ainsworth, lots to read online) that trace the development of this behavior to experiences as an infant. I think these types of people could get stuck in their ways through their 30s and 40s.
[0] http://magazine.uchicago.edu/1012/features/the-nature-of-lon...
Thank you for the description of avoidant personality types, I recognise elements of this in myself.
Similar themes were explored in the anime Neon Genesis Evangelion. I would suggest the main protagonist Shinji definitely has these traits (probably expressed mostly directly through the 'Hedgehog's Dilemma'). I felt a certain amount of relief after finding that show, that at least the creators understood. It also helped that it was a great anime.
To be honest, whilst I don't have the same desire for friendship that I once did, the main thing I do have desire for is a form of sensuality (not sexuality). Moments that are complete without words/with very few words. That's not exactly the easiest thing to communicate to other people, so most of the time I'll just enjoy it in my own time.
When it comes to activities, I'm not short of options. There are new things I want to try, and things I've already enjoyed. What I find less easy is to enjoy them with other people. For example, whilst I've danced with partners before, I tend to prefer dancing by myself. I'm very happy just closing my eyes and letting my body move to the music.
That's almost a useful place to start, as long as you're not being judgmental about it. No one should think lonely people are stuck in a fantasy world.) If you can recognize that something is broken, then you can work on fixing it.
Recognizing that lonely people have fantasy notions of how friendships work is recognizing that in your 30's, it's a bit more work to form a friendship, compared to the lifelong bonds forged by the act of sharing a juice box during recess.
Cognitive behavioral therapy (CBT) is widely used to help reframe and de-fantasize a number of unproductive thought patterns so that instead of fantasizing about a chance meeting where someone instantly becomes your new BFF, mental energy is spent planning and arranging aromantic dates to form the foundation of a friendship, much like trying to find a romantic partner. It's an evidence-based practice for treating mental disorders originally for depression but it's proven helpful in a a wide range of mental health conditions, including identifying upsetting feelings and anxiety that frustrates the a person's attempt to make friends.