I went to a high school that specialised in foreign languages. During my time there I did 6 years of French and German, and 1 year each of Japanese and Spanish. And I can speak none of those. Maybe in German I can just about order a coffee. It's not that I forgot, I just didn't learn anything.
As an adult I have lived in a lot of different countries, including over 12 years in non-English speaking countries. I've been living in my current country for over 5 years, and I speak barely enough to get by in a shop.
Other than my ability to learn languages being non-existent, the issue here is everyone speaks English very well, so it's easy to fall back to it. The other day I went and asked for something in a shop, tried really hard, and the cashier just responded to me in English. In my work everyone speaks English too.
On the other hand, from the time she started speaking, my daughter was fluently able to speak English and her native language. She would come and ask me for something in English, and if I said no, she'd go ask my wife in her language :D
Whenever I pick her up from kindergarden the other kids talk to me in English - they say they learnt by watching YouTube. My colleagues all have similar stories, that they learnt English by watching American TV shows.
I'm guessing what happens is that for a lot of non-native English speakers there is a big advantage to learning English, and once you learn that, your brain figures out how to learn other languages.
But for native English speakers we don't have that, so learning any kind of foreign language is hard. This is probably why countries like Spain or China have low-levels of English speakers (and those that can speak often don't speak well and/or have strong accents), because there isn't much advantage for them to learn it.