I learned Japanese as an adult. I know at least 3 Americans of Japanese descent angry at themselves for complaining to their parents to let them stop extra curricular Japanese classes as children. One was so angry I can't speak an Japanese around him because it just triggers his self loathing over this issue.
I don't have an answer. My dad's side of the family were all professional musicians (though they had other jobs too). My dad never pushed me to play and basically said if I wanted to learn I'd have to push for lessons. Of course I was a kid and just wanted to play video games. I personally wish (or think I wish) they had pushed a little more.