It does cost you to have letters in the alphabet that you don't use (as in, you have minimal pairs with that letter). You don't understand how to pronounce it and it becomes confusing. For example, Italian doesn't use k, j, y, x, w except in loanwords.

And 'z' sound in Italian means two different consonants; in English it also means two different consonants, both also different from both Italian ones. Both are "phonetic".