Except that l/r is pronounced differently than most English speakers, the vowels aren't always pronounced yet take up time ("Nan desu ka" is almost always pronounced like "Nan dess ka", for example), and syllable accent is tone rather than volume for emphasis.
Other than that, it's completely easy. </s>
Just the fact that you can write out romanji and get reasonably understandable Japanese makes it easier than a significant amount of languages out there.
In my opinion the only thing that makes Japanese truly hard is that they decided for some insane reason to borrow the Chinese writing system, which is the most insane writing system in the world, for half (why half!?) of their writing.
I'm approaching this from the angle of wanting to understand video-game Japanese, and when I am trying to parse some 8-bit era all-katakana chunk of text I find myself wishing it included kanji even though I can only recognize a few dozen just so the grammar is more clear.
Moreover [ɾ] can be replaced by [l] perfectly fine and the vowel devoicing is not something you need to do anyway (it's not even occurring in all dialects). So even searching really hard for "difficulties" you can't find even one thing that is remotely difficult which just proves my point.