I think it helps to be married. I can honestly say I'm never lonely as a freelancer (2 years now, I almost never see clients).
It was cool during school breaks to have my older kids at home. Cool during lunch. Not so cool when I had to break up disagreements.
It didn't get really bad for me until I'd been at home 5 or 6 years.