Differentiating between English J and Y is also difficult for Swedes (at least for me), because in Swedish, a J at the start of a word is typically pronounced with a Y-sound. I have to make a conscious effort to make sure I don't get the words jet and yet mixed up when I say them.