Well, growing up in SJC/SV/SF, nonnative speakers were many: roommates, girlfriends, coworkers, and more. Heck, one of my first girlfriends and I could barely communicate verbally: but it wasn't really a problem when gestures, simple words, and "thought synchronization" explained ideas more efficiently and succinctly than learning each others' complex languages.

The other thing is immersion doesn't necessarily seem like a good predictor of language mastery: one roommate had trouble speaking English, despite living in the US for 10 years, while his older brother, who had never been to an English-speaking country before, sounded like he was completely American. This maybe anecdotal but there seems to be an incredible amount of variation in language and accent mastery that doesn't improve with experience (or above a fuzzy, certain age).