It can be both. You can enjoy family time, and still be happy with alone time. I love getting up at 5 am, when everyone is still asleep and the house is still quiet. But I also love spending time with the family once everyone's day starts. Both are possible.

wanting time away from your loved ones doesn't seem the least bit weird to me.

Like sure you love your wife and kids, but being with them every second of every day is probably too much for most sane people. Especially if you have hobbies that demand a lot of focus time.

Have you had children?