Please, don't take this as a personal judgment! I don't know you, I'm just judging what's written here.

> kids activities have to go on a back burner ... I see no reason to place childrens non-needs above critical family infrastructure

This comes across as pretty callous toward your children's development! Social interactions with family are needs for children, too.

It sounds like you chose to go off and build a house unnecessarily and ignored the labor of raising your children to do it, which a lot of people would find pretty irresponsible. So as a rebuttal to someone else saying they can't make the time, because they have work and kids, I think it functions better as an argument against doing what you did. If buying an already-built home wasn't a realistic option for you, that would be very important context here.