It's really no different from immigrant families where the parents don't speak English, but the children speak both languages.

I've known plenty of people and families like that, it was never an issue.