There are many English dialects, and sounds (and meanings) vary, A LOT. It's up to you whether you want to try and assimilate the local dialect of where you live right now, or you want to simply understand and be understood.
There are many non-native speakers who are extremely easy to understand, even though it is clear that English is their second (or third) language. I believe the hardest part is to learn how to make sounds that do not exist in your native language (both consonants and vowels). But the good news is that there are ways to learn that. The human mouth is capable of pronouncing all human sounds, it is only a matter of practice.
The English spelling is guided by meaning, not by sound. So trying to make the sounds out of the letters will always be a frustrating endeavor, as there will never be a single rule you can follow.