You don't have to be around for the whole thing either, you can easily take some time to read or otherwise relax if you have enough friends coming over - nothing weird about friends hanging out in your house without you present.
I used to give out house keys to most of my friends and encourage them to come by whenever - if I'm not around, if I'm busy, or just plain exhausted, I won't hang out, but it's still a safe and fun place for people to gather. This often led to spontaneous 'parties'. I remember once a friend was in the area and came over with ingredients for a dish she had been wanting to create. Enough people ended up randomly showing up, it led to one of the most memorable and wonderful dinner parties of my life!