I've struggled a lot with Mandarin pronunciation. I think the difficulty is mostly due to Chinese speakers being sensitive to differences between some sounds that English speakers lump together.
For example, most English speakers would initially not be aware of any significant difference between the sounds of pinyin "x" and "sh". They're both basically English "sh".
And there are a few other similar cases in Mandarin. Imagine how confused you'd be if someone was speaking English, but randomly swapping the sounds of "p" and "j", and the sounds of "r" and "b", etc.
One thing that makes me love Chinese people even more is that they're so forgiving of my mistakes and accent. The provinces have their own mutually unintelligible dialects, so they're very used to hearing heavy accents. As long as I paint a good enough caricature for them to catch on, it's good enough, and they never make me feel bad about sounding strange. I'm very touched and grateful to them.
Here's something I wrote a while back on this subject: http://nopandas.posthaven.com/understanding-by-listening