You should check out the fourth part of the four hour workweek: filling the void. What you're experiencing is a common issue, and that section has some useful ideas.

You might also consider relocating to a place where people live as you do. In the neighborhood of Montreal I live in, it's very normal for people to work irregular hours, and the streets and cafes are full of people during the workday. I'd feel incredibly isolated in a typical neighborhood that emptied during the 9-5.