That is certainly a factor. But in my personal experience one of the biggest hurdles for Brazilians is the fact that Portuguese have a much shorter vowel pronunciation, so the separation between syllables is much less pronounced. In contrast, some Brazilian accents have comically long vowels, so it sounds like they are spelling out every word.