The person who wrote this blog would disagree with you: www.alljapaneseallthetime.com . It's harder, no doubt, but most people can't give that excuse because they don't go 100% into immersing their lives into the language. Which means listening to audio, TV shows, anything in that language at least 12 hours a day even at work, and even sometimes when you sleep (a lot i know, but if it was that easy, everyone would speak 3+ languages).
Plus, if you live in NYC or any ethnically diverse community in the USA, you'd meet plenty of people who've lived here 20+ years, yet can't have a conversation with you in English - I know many Chinese immigrants like that.