(Going off of memory here, I probably butchered the pronunciation anyway).
/ʒ/ does not occur in word-initial position in English, and for whatever reason it was incredibly hard for me to perceive it there.
There was probably other more subtle differences between the two that did not make it into the transcription (e.g., his native name was pronounced with his native accent instead of his English accent)
If you compare [ta] and [da], you find that the only difference is the time between when you make the consonant, and when your vocal chords start vibrating (voice onset time). In theory, VOT is a contimum, with any value being possible. However, in English it forms a tri-modal distribution /tʰ/ /t/ and /d/. The experiment artificially edited a sound to vary between /t/ and /d/, including with VOTs between the two that do not occur in English. What they found is that people put all of the sounds in 2 boxes, and were unable to distinguish between sounds in the same box, even if their VOT varied considerably.
However, when test subjects were played the same sounds, but told they were listening to rain drops, this effect disapeared, and they were able to distinguish between sounds in the same box.
https://sci-hub.se/https://doi.apa.org/doiLanding?doi=10.103...
Seems to be exactly what you describe.
They also used an ABX methodology, which forces the subject to put sounds into boxes by essentially asking if X is more like A or B, not if X is different from both.
The one I am thinking of used an odd-one-out methodology, where the subject was presented with AAX, and asked to pick which sound was different from the others (where the others were genuinly the same sound). The one I am thinking of also found a priming effect, which yours apparantly didn't.
I never noticed that until now. While it's obviously not an English word, plenty of English speakers have discussed French explorer Jacques Cousteau. I can't think of any other examples.
One of the languages I know allows for a word-initial /ŋ/, which English does not have at all.
Vietnamese, by chance? I'm guessing that because the only initial /ŋ/ word on my radar is the name Nguyen.
Doesn't gendarme start with /ʒ/? Adopted word from French, but almost all words have origins somewhere else.