I don't think that's a reasonable assumption to make. "Your family" could just as easily be referring to adult family members only (spouses, siblings, parents, etc.).
It seems like you're jumping to that conclusion solely because it makes his comments more inflammatory.