Race relations are complicated in the US (and in most of the world). A term that wouldn't seem problematic can become problematic because of its historical context. That's the case with many racial epithets in English...words that seemingly have no negative connotation can take on negative connotation when used for a period of time to insult or separate people based on their race, religion, origin, gender, sexual or gender identity, etc.

No one blames you for not knowing; all you're responsible for is how you respond and correcting future usage. The appropriate response is something along the lines of, "I didn't know. I will say X in the future. Thanks for the heads up." And, if you haven't been told what a better option is, you can generally ask without causing upset.

English is really confusing as a second language. To make it even more complicated, words with racial or other 'insults' change from place to place. Not only that, but the English that is standard through Europe (outside of the UK) is slightly different altogether - folks will be able to talk to other Europeans, but yet get lost when going to an English speaking country.

But no worries, honestly, except for this minor slip, you seem to be doing well.

If someone had made up a fancy word, would that have been better for you? Just accept what people want to be called. Calling black people in the US colored is fraught with problems; I'm sure you wouldn't go to Northern Ireland and call people Fenians.